--- id: "bug-1730" status: "fixed" status_updated: "2026-09-24" status_note: "" component: "controller" severity: "unknown" timestamp: "2026-08-28T13:39:36Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 1 files: ["src/scheduler/src/core_events.c (XCE_CORE_EVENT_JOB_UPDATE case)", "src/controller/sched.c (sched_job_exit_fun/sched_job_start_fun/sched_job_fail_fun)", "src/controller/job.c (HOOK_DETAIL_JOB_EXIT/START/FAIL fire sites)"] tags: ["scheduler", "controller", "job-state", "dead-code", "hook-detail-mismatch", "resource-leak", "accounts-allocs", "raft-apply-decoupling-regression", "tdd", "found-live", "session-sibling-of-bug-1731"] related: ["bug-765", "bug-1679", "bug-1731"] legacy: true --- # bug-1730: User-reported (live 3-node HA cluster, real workload): a finished job's array elements sometimes… ## Description User-reported (live 3-node HA cluster, real workload): a finished job's array elements sometimes remain visible as RUN forever in the scheduler's own state (confirmed both live and via a scheduler session recording's .replay snapshot -- xce_scheduler -i -s showing several 0-12-1787907177-0[N] elements as RUN when the controller's joblist/xcecli already shows them DONE). User's own log correlation: scheduler-side handler.c:414 'sched_handle_sched_update_job updated' fires once per job (the dispatch-time notification) but a second fire for the finish notification never arrives for the affected jobs, while the worker log independently proves those jobs genuinely completed (uv_ipc.c:1042 'Job completed successfully', milliseconds after dispatch -- these are near-instant /bin/true-style array elements). ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction _Not recorded (migrated from buglog.json)._ ## Root cause TWO CONFIRMED FINDINGS from this investigation, one FIXED here, one still OPEN (see bug-1731, its own entry, for the still-open half -- kept separate since the fixed half does not explain the original reported symptom by itself, see below). FINDING 1 (FIXED by this entry): sched_job_exit_fun()/sched_job_start_fun()/sched_job_fail_fun() (controller/sched.c) -- the controller-side notify handlers wired to route a job exit/start/fail specifically to the scheduler's accounts/allocs-cleaning sched_job_state_change() (via NotifyData_data_schedExitJob etc. -> XCE_SCHED_OP_JSTATE_CHANGE) -- are PERMANENTLY DEAD CODE: each only acts on `situation == HOOK_AFTER_EVENT` (default: break for HOOK_BEFORE_EVENT), but HOOK_DETAIL_JOB_EXIT/JOB_START/JOB_FAIL are ONLY EVER fired as HOOK_BEFORE_EVENT anywhere in the controller (job.c:2515/2640/2744/2749/2783, all inside the durable-append-only record_job_*_fact() receipt-time functions -- confirmed via exhaustive grep, zero HOOK_AFTER_EVENT fires of these three details exist). The actual state mutation + notification for a job finish/start/fail happens later, at deferred-apply time (apply_committed_entry()'s shared WAL_JOB_START/FINISH/STOP/RESUME/RUSAGE/CKPT/PREASON/DISPATCH 'group' case, persist.c:4464), which fires a DIFFERENT, generic detail -- HOOK_DETAIL_JOB_UPDATE -- at HOOK_AFTER_EVENT, routing instead through sched_job_update_fun() -> NotifyData_data_schedUpdateJob -> scheduler's sched_handle_sched_update_job() (handler.c:404) -> XCE_SCHED_OP_UPDATE_JOB_FIELDS -> (in the currently-live core_events.c-based driver path) XCE_CORE_EVENT_JOB_UPDATE (core_events.c:933). That handler correctly sets job->state, but -- unlike its sibling XCE_CORE_EVENT_JOB_STATE_CHANGE case right above it in the same file, and unlike scheduler.c's own (also now-dead) sched_job_state_change() -- it never released job->accounts/job->allocs on a terminal transition, so every job finishing via this (the only live) path leaked its resource-allocation/accounting objects for the rest of the scheduler process's life. Almost certainly a regression from the raft-apply-decoupling refactor, which collapsed START/FINISH/STOP/RESUME/RUSAGE/CKPT/PREASON/DISPATCH's deferred-apply hook fire into one generic HOOK_DETAIL_JOB_UPDATE without ever rewiring (or retiring) the scheduler-notification handlers that were built for the old, type-specific hook details. Confirmed this FINDING alone does NOT explain the user's original reported symptom (job->state itself staying RUN): job->state is set correctly by XCE_CORE_EVENT_JOB_UPDATE's existing code regardless of this leak -- the leak only means job->accounts/allocs are never released, a distinct, confirmed-real defect. Extensive live investigation of the actual state-staying-wrong symptom is tracked separately in bug-1731 (open). ## Fix FIXED and TDD-verified (RED->GREEN) for FINDING 1 only. src/scheduler/src/core_events.c's XCE_CORE_EVENT_JOB_UPDATE case: added the identical `if (!JOB_IS_RUN(updated->state)) { free job->accounts via free_job_account(), job->accounts = NULL; free job->allocs via free_JobAlloc_ptr() }` block XCE_CORE_EVENT_JOB_STATE_CHANGE's case already has, right after `job->state = updated->state;`. New test test_job_update_to_terminal_state_frees_accounts_and_allocs (scheduler/testsuite/test_core_events.c): builds a real RUNNING job with a genuine allocation via production make_alloc()/make_host_alloc()/make_job_alloc()/make_job_account() (src/lib/xcealloc.c), pushes a JOB_UPDATE transitioning it to DONE, asserts job->accounts/job->allocs are both NULL afterward. RED confirmed (FAIL: Expected NULL) before the fix; GREEN after. Full scheduler test suite: 52/52 with the new test, only the pre-existing, unrelated symlink-escape-error-message mismatch (sched_admin_request_test) failing before and after -- zero regressions. Real xce_scheduler binary rebuilds cleanly. ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-765, bug-1679, bug-1731