Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
42 lines
5.7 KiB
Markdown
42 lines
5.7 KiB
Markdown
---
|
|
id: "bug-1730"
|
|
status: "fixed"
|
|
status_updated: "2026-09-24"
|
|
status_note: ""
|
|
component: "controller"
|
|
severity: "unknown"
|
|
timestamp: "2026-08-28T13:39:36Z"
|
|
last_seen: "2026-09-19T09:00:00Z"
|
|
occurrences: 1
|
|
files: ["src/scheduler/src/core_events.c (XCE_CORE_EVENT_JOB_UPDATE case)", "src/controller/sched.c (sched_job_exit_fun/sched_job_start_fun/sched_job_fail_fun)", "src/controller/job.c (HOOK_DETAIL_JOB_EXIT/START/FAIL fire sites)"]
|
|
tags: ["scheduler", "controller", "job-state", "dead-code", "hook-detail-mismatch", "resource-leak", "accounts-allocs", "raft-apply-decoupling-regression", "tdd", "found-live", "session-sibling-of-bug-1731"]
|
|
related: ["bug-765", "bug-1679", "bug-1731"]
|
|
legacy: true
|
|
---
|
|
|
|
# bug-1730: User-reported (live 3-node HA cluster, real workload): a finished job's array elements sometimes…
|
|
|
|
## Description
|
|
User-reported (live 3-node HA cluster, real workload): a finished job's array elements sometimes remain visible as RUN forever in the scheduler's own state (confirmed both live and via a scheduler session recording's .replay snapshot -- xce_scheduler -i -s showing several 0-12-1787907177-0[N] elements as RUN when the controller's joblist/xcecli already shows them DONE). User's own log correlation: scheduler-side handler.c:414 'sched_handle_sched_update_job <jobid> updated' fires once per job (the dispatch-time notification) but a second fire for the finish notification never arrives for the affected jobs, while the worker log independently proves those jobs genuinely completed (uv_ipc.c:1042 'Job <id> completed successfully', milliseconds after dispatch -- these are near-instant /bin/true-style array elements).
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Root cause
|
|
TWO CONFIRMED FINDINGS from this investigation, one FIXED here, one still OPEN (see bug-1731, its own entry, for the still-open half -- kept separate since the fixed half does not explain the original reported symptom by itself, see below).
|
|
|
|
FINDING 1 (FIXED by this entry): sched_job_exit_fun()/sched_job_start_fun()/sched_job_fail_fun() (controller/sched.c) -- the controller-side notify handlers wired to route a job exit/start/fail specifically to the scheduler's accounts/allocs-cleaning sched_job_state_change() (via NotifyData_data_schedExitJob etc. -> XCE_SCHED_OP_JSTATE_CHANGE) -- are PERMANENTLY DEAD CODE: each only acts on `situation == HOOK_AFTER_EVENT` (default: break for HOOK_BEFORE_EVENT), but HOOK_DETAIL_JOB_EXIT/JOB_START/JOB_FAIL are ONLY EVER fired as HOOK_BEFORE_EVENT anywhere in the controller (job.c:2515/2640/2744/2749/2783, all inside the durable-append-only record_job_*_fact() receipt-time functions -- confirmed via exhaustive grep, zero HOOK_AFTER_EVENT fires of these three details exist). The actual state mutation + notification for a job finish/start/fail happens later, at deferred-apply time (apply_committed_entry()'s shared WAL_JOB_START/FINISH/STOP/RESUME/RUSAGE/CKPT/PREASON/DISPATCH 'group' case, persist.c:4464), which fires a DIFFERENT, generic detail -- HOOK_DETAIL_JOB_UPDATE -- at HOOK_AFTER_EVENT, routing instead through sched_job_update_fun() -> NotifyData_data_schedUpdateJob -> scheduler's sched_handle_sched_update_job() (handler.c:404) -> XCE_SCHED_OP_UPDATE_JOB_FIELDS -> (in the currently-live core_events.c-based driver path) XCE_CORE_EVENT_JOB_UPDATE (core_events.c:933). That handler correctly sets job->state, but -- unlike its sibling XCE_CORE_EVENT_JOB_STATE_CHANGE case right above it in the same file, and unlike scheduler.c's own (also now-dead) sched_job_state_change() -- it never released job->accounts/job->allocs on a terminal transition, so every job finishing via this (the only live) path leaked its resource-allocation/accounting objects for the rest of the scheduler process's life. Almost certainly a regression from the raft-apply-decoupling refactor, which collapsed START/FINISH/STOP/RESUME/RUSAGE/CKPT/PREASON/DISPATCH's deferred-apply hook fire into one generic HOOK_DETAIL_JOB_UPDATE without ever rewiring (or retiring) the scheduler-notification handlers that were built for the old, type-specific hook details.
|
|
|
|
Confirmed this FINDING alone does NOT explain the user's original reported symptom (job->state itself staying RUN): job->state is set correctly by XCE_CORE_EVENT_JOB_UPDATE's existing code regardless of this leak -- the leak only means job->accounts/allocs are never released, a distinct, confirmed-real defect. Extensive live investigation of the actual state-staying-wrong symptom is tracked separately in bug-1731 (open).
|
|
|
|
## Fix
|
|
FIXED and TDD-verified (RED->GREEN) for FINDING 1 only. src/scheduler/src/core_events.c's XCE_CORE_EVENT_JOB_UPDATE case: added the identical `if (!JOB_IS_RUN(updated->state)) { free job->accounts via free_job_account(), job->accounts = NULL; free job->allocs via free_JobAlloc_ptr() }` block XCE_CORE_EVENT_JOB_STATE_CHANGE's case already has, right after `job->state = updated->state;`. New test test_job_update_to_terminal_state_frees_accounts_and_allocs (scheduler/testsuite/test_core_events.c): builds a real RUNNING job with a genuine allocation via production make_alloc()/make_host_alloc()/make_job_alloc()/make_job_account() (src/lib/xcealloc.c), pushes a JOB_UPDATE transitioning it to DONE, asserts job->accounts/job->allocs are both NULL afterward. RED confirmed (FAIL: Expected NULL) before the fix; GREEN after. Full scheduler test suite: 52/52 with the new test, only the pre-existing, unrelated symlink-escape-error-message mismatch (sched_admin_request_test) failing before and after -- zero regressions. Real xce_scheduler binary rebuilds cleanly.
|
|
|
|
## Verification
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## References
|
|
- Related: bug-765, bug-1679, bug-1731
|