lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler-sim-reconcile-terminal-guard): task 1.2 - bug-1764 terminal-state guard

Browse source 
sim_reconcile_unaccounted_jobs()'s per-job trigger check (sim_loop.c)
now also excludes JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state),
alongside the existing job->accounts != NULL skip. A terminal job is
never searched for a matching recorded DISPATCH entry, so a leftover
or duplicate DISPATCH decision from a later replay round can no
longer resurrect allocs/accounts on a job that has already completed
- the exact mechanism behind info worker's assigned-jobs listing
showing a finished job as still consuming a worker's resources.

The exclusion mirrors job_state_regresses_terminal()'s existing
"terminal is permanent" invariant (core_events.c) rather than
inventing a new rule. The function's doc comment gains a paragraph
clarifying the terminal exclusion is complementary to, not a reversal
of, the existing PEND/RUN state-independence rationale that motivated
this trigger's original widening.

Both RED-phase tests (task 1.1) pass; the pre-existing PEND
eager-capture regression test and core_events.c's JOBS_PUSH
resync-merge terminal-state tests are unaffected. The analogous
resync-merge branch was investigated and confirmed already correct.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
This commit is contained in:
Rongsong Shen 2026-09-03 07:15:42 +00:00
parent ff05c37bca
commit b722de4f08
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
3 changed files with 72 additions and 23 deletions
Download patch file Download diff file Expand all files Collapse all files

27
.wolf/buglog.json
View file

@ -20801,6 +20801,31 @@
],
"occurrences": 1,
"last_seen": "2026-09-02T13:00:00Z"
},
{
"id": "bug-1764",
"timestamp": "2026-09-03T03:01:26Z",
"error_message": "info worker's \"assigned jobs\" listing (repl_info_worker_one(), repl_info.c) can show a job that has already reached a terminal state (DONE/EXIT) as still assigned to a worker, contradicting info summary's running count. Live-observed in a real interactive session (session=59): info summary reported \"100 jobs (0 pending, 7 running)\" and worker pc13v's slots showed used=7.00, but the same info worker output listed assigned jobs: 0-11-1788164699-0[0-49] -- the full 50-index range of one array job, all still shown assigned.",
"file": "src/scheduler/src/sim_loop.c",
"root_cause": "sim_reconcile_unaccounted_jobs() (sim_loop.c:664) triggers reconciliation on job->accounts == NULL alone (sim_loop.c:690), with no check on job->state. This trigger was deliberately widened to be state-independent by the scheduler-sim-eager-reconciliation spec (task 1.1, test_sim_reconcile_unaccounted.c's own Test 5 header comment: \"capture eligibility no longer depends on job->state\") so a PEND-state job's early-recorded DISPATCH decision could be captured before its RUN confirmation replays. That widening only considered the forward case (PEND -> captured early -> later RUN) and missed the terminal case: a job that already ran to completion has job->accounts/job->allocs correctly freed by xce_core_push_events()'s normal end-of-life cleanup (core_events.c:1127-1135 JOB_STATE_CHANGE, :1179-1186 JOB_UPDATE) -- which also leaves job->accounts == NULL. If ANY later round's decoded_session->decisions still carries an unconsumed DISPATCH entry for that same jobid (plausible in a real recording with duplicate/retried dispatch attempts, or per-round decision-emission timing not lining up cleanly with per-job completion for an array job), sim_reconcile_unaccounted_jobs() matches it and resurrects job->allocs/job->accounts on the already-terminal job, with no state check to prevent it. Dynamically confirmed via a temporary probe test (added to test_sim_reconcile_unaccounted.c, run once, then reverted -- never committed): a job driven to JOB_STAT_DONE via xce_core_push_events() (which correctly nulled its allocs/accounts) was then fed a second decoded_session containing a fresh DISPATCH entry for the same jobid; sim_reconcile_unaccounted_jobs() unconditionally re-populated both job->allocs and job->accounts, contradicting job->state == JOB_STAT_DONE. repl_info_worker_one()'s assigned-jobs filter (repl_info.c:686, if (job->allocs == NULL) continue;) has no independent state check either, so it faithfully (and misleadingly) displays the resurrected allocation. The already-established defensive pattern for exactly this class of problem, job_state_regresses_terminal() (core_events.c:696-701, \"terminal states are permanent\"), is not reused here.",
"fix": "FIXED (2026-09-03, scheduler-sim-reconcile-terminal-guard spec, task 1.2). sim_reconcile_unaccounted_jobs()'s per-job trigger check (sim_loop.c) now also excludes JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state), alongside the existing job->accounts != NULL skip -- a terminal job is never searched for a matching recorded DISPATCH entry, mirroring job_state_regresses_terminal()'s existing 'terminal is permanent' invariant (core_events.c). TDD: task 1.1's RED-phase tests (test_reconcile_never_reallocates_a_done_job_despite_stray_dispatch_entry, test_reconcile_never_reallocates_an_exit_job_despite_stray_dispatch_entry, test_sim_reconcile_unaccounted.c) pinned the exact reproduction before the fix landed; both pass after. Confirmed no regression to the pre-existing PEND eager-capture case or to core_events.c's JOBS_PUSH resync-merge terminal-state tests. The analogous resync-merge branch was investigated and confirmed already correct (job_state_regresses_terminal() call), no change needed there.",
"tags": [
"scheduler",
"sim",
"repl",
"reconciliation",
"job-lifecycle",
"stale-state",
"array-job"
],
"related_bugs": [
"bug-1611",
"bug-1734",
"bug-1738",
"bug-1739"
],
"occurrences": 1,
"last_seen": "2026-09-03T03:01:26Z"
}
]
}
}