Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
4.9 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1738 | fixed | 2026-09-24 | scheduler | unknown | 2026-08-30T22:10:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1738: Found investigating Symptom 4 (EXIT vs DONE divergence) alongside bug-1737: a SEPARATE, genuinely…
Description
Found investigating Symptom 4 (EXIT vs DONE divergence) alongside bug-1737: a SEPARATE, genuinely different mechanism that can ALSO clobber a correct terminal job state on the scheduler side, independent of the schedExitJob exit_code gap. Not separately live-reproduced this session (no post_exec-stage job failure was exercised against the live cluster) -- confirmed by direct code reading with an exact, unambiguous mechanism.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
CONFIRMED via direct code reading. HOOK_DETAIL_JOB_FAIL is a narrower event than HOOK_DETAIL_JOB_EXIT/finish -- job.c's apply_job_fail_transition() only forces JOB_STAT_EXIT for a genuinely pre-exit failure (the default: case, no prior exit info exists yet); for JOB_EXEC_STAGE_POST_EXEC/Q_POST_EXEC (a post-exec script failing on an otherwise-already-decided job) it MERGES a JOB_STAT_PERR flag onto whatever DONE/EXIT the job's own finish already decided -- it never re-decides DONE vs EXIT after the fact. But sched.c's sched_job_fail_fun() (the HOOK_DETAIL_JOB_FAIL subscriber) always sends the SAME bare-jobid schedFailJob notify regardless of which case applied, and scheduler/src/handler.c's sched_handle_notify() unconditionally maps THAT to JOB_STAT_EXIT via XCE_CORE_EVENT_JOB_STATE_CHANGE, which (before this fix) had NO guard against overwriting an already-terminal state: job->state = new_state; unconditionally. Net effect: a job that finished successfully (DONE, correctly notified live) and then has an UNRELATED post-exec script fail afterward would have the scheduler's view of it forcibly clobbered from DONE back to EXIT -- a real, distinct correctness bug from bug-1737's exit_code gap. SEPARATELY, this session's own bug-1734 fix (landed earlier the same session, src/scheduler/src/core_events.c, XCE_CORE_EVENT_JOBS_PUSH's duplicate-jobid merge path) has the IDENTICAL class of gap: on a resync redelivery carrying a DIFFERENT state than the existing tracked job, it merged existing->state = job->state unconditionally with no ordering/terminal-permanence check -- a stale or out-of-order resync redelivery claiming EXIT for a job already correctly DONE would silently regress it. Both gaps share one missing invariant: once a job reaches a terminal state (DONE or EXIT) on the scheduler's own tracked copy, that state must be PERMANENT -- mirroring the controller's own job_state_is_terminal() doctrine (job.c: apply_job_finish_transition() no-ops a duplicate FINISH once terminal) -- and no single-job notify or resync redelivery should ever be able to change it to something DIFFERENT.
Fix
FIXED (2026-08-30). Added one shared static helper, job_state_regresses_terminal(current_state, new_state) (core_events.c, just above xce_core_push_events()): true iff current_state is already DONE or EXIT AND new_state is different. Applied it at BOTH vulnerable call sites: (1) XCE_CORE_EVENT_JOB_STATE_CHANGE's case -- rejects (WARN-logged, no-op) a new_state that would regress an already-terminal job, before the existing accounts/allocs-cleanup + state-assignment logic runs; (2) the JOBS_PUSH duplicate-jobid merge path (bug-1734) -- same guard, applied before deciding how to merge. Also added a clarifying comment to sched.c's sched_job_fail_fun() documenting the POST_EXEC-vs-default distinction and why this core-side guard (not a sched.c-side fix) is the right place to close the gap, since the scheduler has no JOB_STAT_PERR-equivalent concept of its own to merge instead -- 'never regress a terminal state' is simpler and sufficient. TDD: added test_job_state_change_does_not_regress_terminal_state (drives a job to DONE via JOB_STATE_CHANGE, then a second JOB_STATE_CHANGE claiming EXIT must be ignored) and test_jobs_push_duplicate_jobid_does_not_regress_terminal_state (same invariant via the JOBS_PUSH merge path) in test_core_events.c. Full scheduler+controller suite: 61/61 Ok, zero regressions -- the pre-existing bug-1734 test (PEND -> DONE merge, a legitimate forward transition) still passes unchanged, confirming the new guard only blocks a REGRESSION (terminal -> different), not a legitimate non-terminal -> terminal merge.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1734, bug-1737, bug-1739