Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
6 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1811 | fixed | 2026-09-24 | scheduler | unknown | 2026-09-15T07:40:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1811: User-reported (interactive sim session=35, info jobs/info worker pc13v): job…
Description
User-reported (interactive sim session=35, info jobs/info worker pc13v): job 0-14-1789369200-0[21] shows RUN in info jobs but is absent from info worker pc13v's assigned-jobs listing (which lists [0-20,22], skipping 21) -- user's working theory was slot overcommitment (23 running jobs vs. 22-slot worker capacity).
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
NOT A BUG -- initial diagnosis was WRONG and the attempted fix was reverted. Root-caused (correctly, mechanism-wise) to XCE_CORE_EVENT_JOB_STATE_CHANGE/XCE_CORE_EVENT_JOB_UPDATE (core_events.c) setting job->state=RUN unconditionally even when job->allocs/job->accounts are still NULL -- repl_info_worker_one() (repl_info.c) walks job->allocs to build the assigned-jobs listing, so an unaccounted RUN job is correctly displayed as RUN in info jobs but silently absent from info worker's listing. Initially misdiagnosed this AS a defect (a 'phantom RUN' analogous to bug-1739's JOBS_PUSH resync gap) and implemented a guard refusing the RUN transition without an existing allocation. That guard was WRONG: this exact 'RUN with allocs==NULL' transient window is the INTENTIONAL, designed, tested trigger condition for Recorded-Decision Reconciliation (scheduler-sim-decision-parity spec, sim_loop.c's sim_reconcile_unaccounted_jobs(), extended by bug-1804's 2026-09-11 revision to set job->state=RUN synchronously via JOB_STATE_CHANGE/JOB_UPDATE ahead of allocation capture, matching the original live scheduler's own sched_commit_decision() semantics). Confirmed via test_sim_e2e_original_motivation.c's own header comment and ANCHOR_ID fixture: the SAME core_events.c code path is used, by design, to construct a job that is 'already running, with an unaccounted allocation, when a scheduling round's own live decision-making reaches it' -- 'Recorded-Decision Reconciliation's entire reason for existing'. My guard broke this fixture (test_e2e_original_motivation_long_replay) plus test_push_events_fires_each_hook_once_per_event_with_picker_idle and test_drain_translates_and_pushes_all_ten_op_types, both of which legitimately exercise a bare JOB_STATE_CHANGE/JOB_UPDATE(RUN) with no allocation yet. Further confirmed against scheduler-sim-decision-parity/requirements.md Requirement 4 ('Graceful degradation when recorded decision data is unavailable'): AC 4.1/4.2 EXPLICITLY anticipate and accept that a job's originating DISPATCH decision may not be found in the recorded replay data, in which case the scheduler reports that job's usage as unavailable rather than fabricating a value, and continues processing -- this is precisely job 21's observed symptom, not an accident. The user's overcommitment theory is also independently wrong on the accounting side: sched_account_jobs() only charges resInUse for JOB_IS_RUN(state) && accounts != NULL, so an unaccounted RUN job is correctly EXCLUDED from the real slot-usage total (used=22.00 already correctly reflects only the 22 genuinely-accounted jobs) -- there is no real overcommitment, only a display gap for a job whose allocation reconciliation hasn't (yet, or ever, depending on whether its DISPATCH decision exists later in this replay stream) caught up.
Fix
REVERTED. Initial fix (guard in core_events.c's XCE_CORE_EVENT_JOB_STATE_CHANGE/JOB_UPDATE refusing a RUN claim without an existing allocation, plus 6 new tests in test_core_events.c and a fixture change in test_driver_translate.c) was implemented, RED/GREEN-tested, then found to break test_sim_e2e_original_motivation.c's test_e2e_original_motivation_long_replay (the live-verify for the entire Recorded-Decision Reconciliation mechanism) plus two pre-existing test_core_events.c/test_driver_translate.c tests. All changes reverted via git checkout -- before any commit. No production code changed. Left as an open question whether job 21 (and the user's follow-up observation that this 'occurs more frequently') ever actually reconciles later in the same replay session -- if a job's RUN-but-unaccounted display state NEVER resolves even after the replay completes, that would indicate its DISPATCH decision is genuinely missing from the recorded stream (the same category of gap scheduler-snapshot-job-allocation's PREEXIST_ID addresses for pre-recording dispatches, via snapshot-based restoration instead of replay-based reconciliation) -- worth a follow-up investigation with concrete evidence (does the job ever show up in info worker's listing on a later round?) before attempting any further fix. A safe, smaller follow-up actually worth considering: improve info worker's assigned-jobs display (repl_info.c) to explicitly distinguish a RUN-but-unaccounted job (e.g. an annotation) from silent omission, so operators aren't misled into an overcommitment theory the way this session's user was -- proposed to the user, not yet implemented. SUPERSEDED (2026-09-15, same session): the 'not a bug' conclusion above was itself only half right. The transient RUN-before-allocation window IS legitimate (Recorded-Decision Reconciliation), but the specific PERMANENT loss the user kept observing turned out to be a real, separate defect in the JOBS_PUSH duplicate-jobid resync merge -- see bug-1812 for the actual root cause and fix.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1739, bug-1804, bug-1778