Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
5.7 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1730 | fixed | 2026-09-24 | controller | unknown | 2026-08-28T13:39:36Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1730: User-reported (live 3-node HA cluster, real workload): a finished job's array elements sometimes…
Description
User-reported (live 3-node HA cluster, real workload): a finished job's array elements sometimes remain visible as RUN forever in the scheduler's own state (confirmed both live and via a scheduler session recording's .replay snapshot -- xce_scheduler -i -s showing several 0-12-1787907177-0[N] elements as RUN when the controller's joblist/xcecli already shows them DONE). User's own log correlation: scheduler-side handler.c:414 'sched_handle_sched_update_job updated' fires once per job (the dispatch-time notification) but a second fire for the finish notification never arrives for the affected jobs, while the worker log independently proves those jobs genuinely completed (uv_ipc.c:1042 'Job completed successfully', milliseconds after dispatch -- these are near-instant /bin/true-style array elements).
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
TWO CONFIRMED FINDINGS from this investigation, one FIXED here, one still OPEN (see bug-1731, its own entry, for the still-open half -- kept separate since the fixed half does not explain the original reported symptom by itself, see below).
FINDING 1 (FIXED by this entry): sched_job_exit_fun()/sched_job_start_fun()/sched_job_fail_fun() (controller/sched.c) -- the controller-side notify handlers wired to route a job exit/start/fail specifically to the scheduler's accounts/allocs-cleaning sched_job_state_change() (via NotifyData_data_schedExitJob etc. -> XCE_SCHED_OP_JSTATE_CHANGE) -- are PERMANENTLY DEAD CODE: each only acts on situation == HOOK_AFTER_EVENT (default: break for HOOK_BEFORE_EVENT), but HOOK_DETAIL_JOB_EXIT/JOB_START/JOB_FAIL are ONLY EVER fired as HOOK_BEFORE_EVENT anywhere in the controller (job.c:2515/2640/2744/2749/2783, all inside the durable-append-only record_job_*_fact() receipt-time functions -- confirmed via exhaustive grep, zero HOOK_AFTER_EVENT fires of these three details exist). The actual state mutation + notification for a job finish/start/fail happens later, at deferred-apply time (apply_committed_entry()'s shared WAL_JOB_START/FINISH/STOP/RESUME/RUSAGE/CKPT/PREASON/DISPATCH 'group' case, persist.c:4464), which fires a DIFFERENT, generic detail -- HOOK_DETAIL_JOB_UPDATE -- at HOOK_AFTER_EVENT, routing instead through sched_job_update_fun() -> NotifyData_data_schedUpdateJob -> scheduler's sched_handle_sched_update_job() (handler.c:404) -> XCE_SCHED_OP_UPDATE_JOB_FIELDS -> (in the currently-live core_events.c-based driver path) XCE_CORE_EVENT_JOB_UPDATE (core_events.c:933). That handler correctly sets job->state, but -- unlike its sibling XCE_CORE_EVENT_JOB_STATE_CHANGE case right above it in the same file, and unlike scheduler.c's own (also now-dead) sched_job_state_change() -- it never released job->accounts/job->allocs on a terminal transition, so every job finishing via this (the only live) path leaked its resource-allocation/accounting objects for the rest of the scheduler process's life. Almost certainly a regression from the raft-apply-decoupling refactor, which collapsed START/FINISH/STOP/RESUME/RUSAGE/CKPT/PREASON/DISPATCH's deferred-apply hook fire into one generic HOOK_DETAIL_JOB_UPDATE without ever rewiring (or retiring) the scheduler-notification handlers that were built for the old, type-specific hook details.
Confirmed this FINDING alone does NOT explain the user's original reported symptom (job->state itself staying RUN): job->state is set correctly by XCE_CORE_EVENT_JOB_UPDATE's existing code regardless of this leak -- the leak only means job->accounts/allocs are never released, a distinct, confirmed-real defect. Extensive live investigation of the actual state-staying-wrong symptom is tracked separately in bug-1731 (open).
Fix
FIXED and TDD-verified (RED->GREEN) for FINDING 1 only. src/scheduler/src/core_events.c's XCE_CORE_EVENT_JOB_UPDATE case: added the identical if (!JOB_IS_RUN(updated->state)) { free job->accounts via free_job_account(), job->accounts = NULL; free job->allocs via free_JobAlloc_ptr() } block XCE_CORE_EVENT_JOB_STATE_CHANGE's case already has, right after job->state = updated->state;. New test test_job_update_to_terminal_state_frees_accounts_and_allocs (scheduler/testsuite/test_core_events.c): builds a real RUNNING job with a genuine allocation via production make_alloc()/make_host_alloc()/make_job_alloc()/make_job_account() (src/lib/xcealloc.c), pushes a JOB_UPDATE transitioning it to DONE, asserts job->accounts/job->allocs are both NULL afterward. RED confirmed (FAIL: Expected NULL) before the fix; GREEN after. Full scheduler test suite: 52/52 with the new test, only the pre-existing, unrelated symlink-escape-error-message mismatch (sched_admin_request_test) failing before and after -- zero regressions. Real xce_scheduler binary rebuilds cleanly.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-765, bug-1679, bug-1731