diff --git a/.kiro/specs/scheduler-sim-reconcile-terminal-guard/tasks.md b/.kiro/specs/scheduler-sim-reconcile-terminal-guard/tasks.md index 6590dc3d..10f808a0 100644 --- a/.kiro/specs/scheduler-sim-reconcile-terminal-guard/tasks.md +++ b/.kiro/specs/scheduler-sim-reconcile-terminal-guard/tasks.md @@ -7,7 +7,7 @@ - _Requirements: 1.1, 1.2, 1.3_ - _Boundary: Terminal-State Guard_ -- [ ] 1.2 Implement the terminal-state exclusion in sim_reconcile_unaccounted_jobs() +- [x] 1.2 Implement the terminal-state exclusion in sim_reconcile_unaccounted_jobs() - Add JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state) as an additional exclusion in the function's existing per-job trigger check, alongside the existing job->accounts != NULL skip; update the function's own doc comment to state the terminal exclusion alongside its existing state-independence rationale so the two don't read as contradictory - Observable: all tests from 1.1 pass; a terminal job's allocs/accounts are provably unchanged before and after a call that presents a matching stray DISPATCH entry for it - _Requirements: 1.1, 1.2, 1.3_ diff --git a/.wolf/buglog.json b/.wolf/buglog.json index ddbc10ac..e2a5920c 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -20801,6 +20801,31 @@ ], "occurrences": 1, "last_seen": "2026-09-02T13:00:00Z" + }, + { + "id": "bug-1764", + "timestamp": "2026-09-03T03:01:26Z", + "error_message": "info worker's \"assigned jobs\" listing (repl_info_worker_one(), repl_info.c) can show a job that has already reached a terminal state (DONE/EXIT) as still assigned to a worker, contradicting info summary's running count. Live-observed in a real interactive session (session=59): info summary reported \"100 jobs (0 pending, 7 running)\" and worker pc13v's slots showed used=7.00, but the same info worker output listed assigned jobs: 0-11-1788164699-0[0-49] -- the full 50-index range of one array job, all still shown assigned.", + "file": "src/scheduler/src/sim_loop.c", + "root_cause": "sim_reconcile_unaccounted_jobs() (sim_loop.c:664) triggers reconciliation on job->accounts == NULL alone (sim_loop.c:690), with no check on job->state. This trigger was deliberately widened to be state-independent by the scheduler-sim-eager-reconciliation spec (task 1.1, test_sim_reconcile_unaccounted.c's own Test 5 header comment: \"capture eligibility no longer depends on job->state\") so a PEND-state job's early-recorded DISPATCH decision could be captured before its RUN confirmation replays. That widening only considered the forward case (PEND -> captured early -> later RUN) and missed the terminal case: a job that already ran to completion has job->accounts/job->allocs correctly freed by xce_core_push_events()'s normal end-of-life cleanup (core_events.c:1127-1135 JOB_STATE_CHANGE, :1179-1186 JOB_UPDATE) -- which also leaves job->accounts == NULL. If ANY later round's decoded_session->decisions still carries an unconsumed DISPATCH entry for that same jobid (plausible in a real recording with duplicate/retried dispatch attempts, or per-round decision-emission timing not lining up cleanly with per-job completion for an array job), sim_reconcile_unaccounted_jobs() matches it and resurrects job->allocs/job->accounts on the already-terminal job, with no state check to prevent it. Dynamically confirmed via a temporary probe test (added to test_sim_reconcile_unaccounted.c, run once, then reverted -- never committed): a job driven to JOB_STAT_DONE via xce_core_push_events() (which correctly nulled its allocs/accounts) was then fed a second decoded_session containing a fresh DISPATCH entry for the same jobid; sim_reconcile_unaccounted_jobs() unconditionally re-populated both job->allocs and job->accounts, contradicting job->state == JOB_STAT_DONE. repl_info_worker_one()'s assigned-jobs filter (repl_info.c:686, if (job->allocs == NULL) continue;) has no independent state check either, so it faithfully (and misleadingly) displays the resurrected allocation. The already-established defensive pattern for exactly this class of problem, job_state_regresses_terminal() (core_events.c:696-701, \"terminal states are permanent\"), is not reused here.", + "fix": "FIXED (2026-09-03, scheduler-sim-reconcile-terminal-guard spec, task 1.2). sim_reconcile_unaccounted_jobs()'s per-job trigger check (sim_loop.c) now also excludes JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state), alongside the existing job->accounts != NULL skip -- a terminal job is never searched for a matching recorded DISPATCH entry, mirroring job_state_regresses_terminal()'s existing 'terminal is permanent' invariant (core_events.c). TDD: task 1.1's RED-phase tests (test_reconcile_never_reallocates_a_done_job_despite_stray_dispatch_entry, test_reconcile_never_reallocates_an_exit_job_despite_stray_dispatch_entry, test_sim_reconcile_unaccounted.c) pinned the exact reproduction before the fix landed; both pass after. Confirmed no regression to the pre-existing PEND eager-capture case or to core_events.c's JOBS_PUSH resync-merge terminal-state tests. The analogous resync-merge branch was investigated and confirmed already correct (job_state_regresses_terminal() call), no change needed there.", + "tags": [ + "scheduler", + "sim", + "repl", + "reconciliation", + "job-lifecycle", + "stale-state", + "array-job" + ], + "related_bugs": [ + "bug-1611", + "bug-1734", + "bug-1738", + "bug-1739" + ], + "occurrences": 1, + "last_seen": "2026-09-03T03:01:26Z" } ] -} +} \ No newline at end of file diff --git a/src/scheduler/src/sim_loop.c b/src/scheduler/src/sim_loop.c index 020b7c24..ce9ccdff 100644 --- a/src/scheduler/src/sim_loop.c +++ b/src/scheduler/src/sim_loop.c @@ -586,26 +586,49 @@ static void* sim_picker_thread_fn(void* arg) * site's own comment below. Still directly unit-testable standalone too * (test_sim_reconcile_unaccounted.c), per design.md's own Testing Strategy. * - * Trigger condition (scheduler-sim-eager-reconciliation spec, task 1.1): - * `job->accounts == NULL` is now the SOLE capture-eligibility gate -- - * state-independent, reused here as both the trigger AND (now that the - * FOUND branch below populates job->accounts) the double-application guard. - * Any job in `sim->core.jobs` with `job->accounts == NULL` is eligible to be - * searched for a matching recorded DISPATCH decision this round, regardless - * of its current `job->state` -- a job's recorded DISPATCH decision and its - * RUN-state confirmation can legitimately land many rounds apart in the - * original recording (the dominant real-world case), and this function is - * called unconditionally every round, so a not-yet-running job is simply - * re-checked round after round until its originating decision is found (or - * the replay ends). Actual resource charging remains fully deferred to - * sched_account_jobs()'s own unchanged `JOB_IS_RUN` gate (scheduler.c) -- - * capturing a job's allocation here while it is not yet RUN is inert until - * that gate later includes it; this function performs no charging itself. - * Requirement 3.3's "report only the fresh one" guarantee needs no separate - * mechanism here: a job that independently receives a fresh live decision in - * this same round already has job->accounts set by sched_commit_decision() - * (scheduler.c, unchanged) by the time this trigger check runs, so this - * function simply finds nothing to do for it. + * Trigger condition (scheduler-sim-eager-reconciliation spec, task 1.1; + * narrowed by scheduler-sim-reconcile-terminal-guard task 1.2, bug-1764): + * `job->accounts == NULL` is the primary capture-eligibility gate -- + * state-independent WITH RESPECT TO PEND vs. RUN, reused here as both the + * trigger AND (now that the FOUND branch below populates job->accounts) the + * double-application guard. Any job in `sim->core.jobs` with `job->accounts + * == NULL` is eligible to be searched for a matching recorded DISPATCH + * decision this round, regardless of whether it is PEND or RUN -- a job's + * recorded DISPATCH decision and its RUN-state confirmation can legitimately + * land many rounds apart in the original recording (the dominant real-world + * case), and this function is called unconditionally every round, so a + * not-yet-running job is simply re-checked round after round until its + * originating decision is found (or the replay ends). Actual resource + * charging remains fully deferred to sched_account_jobs()'s own unchanged + * `JOB_IS_RUN` gate (scheduler.c) -- capturing a job's allocation here while + * it is not yet RUN is inert until that gate later includes it; this + * function performs no charging itself. Requirement 3.3's "report only the + * fresh one" guarantee needs no separate mechanism here: a job that + * independently receives a fresh live decision in this same round already + * has job->accounts set by sched_commit_decision() (scheduler.c, unchanged) + * by the time this trigger check runs, so this function simply finds + * nothing to do for it. + * + * This PEND/RUN state-independence does NOT extend to a job that has + * already reached a terminal state: `JOB_IS_DONE(job->state) || + * JOB_IS_EXIT(job->state)` is an additional, unconditional exclusion, + * checked alongside `job->accounts != NULL` in the same trigger `if`. A + * terminal job's `job->accounts`/`job->allocs` are freed by + * xce_core_push_events()'s own end-of-life cleanup on the terminal + * transition (core_events.c's JOB_STATE_CHANGE/JOB_UPDATE cases), which + * leaves `job->accounts == NULL` -- the exact same shape as a legitimately + * not-yet-captured PEND/RUN job. Without this exclusion, a later round's + * decoded_session carrying a stale/duplicate DISPATCH entry for that same + * jobid (e.g. a retried dispatch attempt for one element of an array job) + * would be indistinguishable from a genuine early-dispatch case and would + * resurrect the allocation on an already-dead job (bug-1764). The two rules + * compose without contradiction: state-independence is about WHEN a + * (still-live) job's own DISPATCH decision happens to replay relative to + * its RUN confirmation -- it says nothing about a job that can never + * legitimately be reconsidered again because it has already finished. This + * mirrors job_state_regresses_terminal()'s (core_events.c) existing + * "terminal is permanent" invariant, applied here at the allocation layer + * instead of the state-transition layer. * * For each such job, searches `decoded_session->decisions` for a matching * DISPATCH entry by jobid -- a job can only ever be captured via a recorded @@ -687,7 +710,8 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim, break; } - if ((job == NULL) || (job->accounts != NULL)) { + if ((job == NULL) || (job->accounts != NULL) || JOB_IS_DONE(job->state) + || JOB_IS_EXIT(job->state)) { continue; }