Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
5.4 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1739 | fixed | 2026-09-24 | scheduler | unknown | 2026-08-30T22:15:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1739: User-reported (live 3-node HA cluster + worker, 50-element array job): info summary showed 30…
Description
User-reported (live 3-node HA cluster + worker, 50-element array job): info summary showed 30 elements of an array marked RUN in info jobs, but info worker pc13v only listed 22 'assigned job:' lines (matching the worker's real 22-core capacity) -- 8 elements showed job-level state RUN with NO corresponding worker/slot-assignment record at all, a 'phantom RUN' exceeding real hardware capacity.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
CONFIRMED via direct code reading (this exact mechanism was identified and fixed before a dedicated live reproduction of the PRE-fix phantom-RUN state was attempted; the FIX itself was verified live -- a real 50-element array job run against the fixed binary showed zero phantom RUN jobs at any inspection point). This session's own bug-1734 fix (landed earlier the same session) added a merge for XCE_CORE_EVENT_JOBS_PUSH's duplicate-jobid (EEXIST) case: on a resync redelivery carrying a DIFFERENT state than the existing tracked job, if the new state was RUN, it set existing->state = job->state (RUN) directly -- but did NOT transfer any allocation data. core_build_job() (this same file, called earlier in the SAME event to construct the redelivered job clone from the wire xce_job_t) already adopts the wire payload's allocs field into a fresh xce_sched_job_t exactly the same way a brand-new RUN job's JOBS_PUSH would (job->accounts = make_job_account(job->allocs) when j->allocs != NULL) -- i.e. the redelivered clone DOES carry real allocation data when the controller's resync wire payload includes it (xcesched_push_jobs(), sched.c, collects the controller's live job_t structs directly, whose ->job.allocs is the real dispatch record for a running job). But bug-1734's merge branch discarded this clone entirely (FREEUP(job) at the end of the EEXIST handler, with allocs/accounts explicitly left out of scope in that fix's own comment) without ever copying job->allocs/job->accounts onto existing -- leaving existing->state == RUN with existing->allocs == NULL: a job the scheduler's own resource/slot bookkeeping (info worker's 'assigned job:' scan, which walks core->jobs' allocations) has no record of at all, exactly the reported phenotype.
Fix
FIXED (2026-08-30), same edit pass as bug-1738's core_events.c guard (both touch the same JOBS_PUSH EEXIST branch). When the merge's terminal-permanence guard (bug-1738) allows the merge to proceed and the redelivered state is RUN: if the redelivered clone carries real allocation data (job->accounts != NULL && job->allocs != NULL), TRANSFER it onto existing (free whatever stale accounts/allocs existing had, then move the pointers over -- existing->accounts = job->accounts; existing->allocs = job->allocs; job->accounts = NULL; job->allocs = NULL; so the later FREEUP(job) doesn't double-free them) instead of leaving them untouched/discarded. If the redelivery claims RUN but carries NO allocation data (a defensive case -- an older/degraded controller payload, or a race where the dispatch record wasn't attached yet), the merge into RUN is REFUSED (WARN-logged, existing state kept unchanged) rather than manufacturing a phantom RUN from a different cause. TDD: added test_jobs_push_duplicate_jobid_run_state_transfers_allocation (RUN + real allocs -> existing->state==RUN AND existing->allocs/accounts non-NULL, transferred not discarded) and test_jobs_push_duplicate_jobid_run_state_without_allocation_is_refused (RUN + no allocs -> existing stays at its prior safe state) in test_core_events.c. Full scheduler+controller suite: 61/61 Ok, zero regressions -- bug-1734's own original test (PEND -> DONE merge, no allocation involved) still passes unchanged. LIVE RE-VERIFICATION: rebuilt, redeployed a fresh 3-node HA cluster + 22-core worker, submitted a 50-element overlapping-runtime array job (sleep 60 && hostname) plus one standalone job against the 22-slot capacity, captured a mid-run scheduler recording (19 jobs genuinely RUN, matching the worker's real capacity minus a few still-queued elements) and inspected info worker/info jobs: RUN-state job count matched 'assigned job:' count exactly at every inspection point (23 assigned = 19 RUN + 4 jobs with an in-flight allocation reservation not yet finalized to RUN, a benign timing artifact of catching mid-dispatch state, not a phantom RUN) -- no job ever showed RUN with a missing worker assignment. Could not force the SAME reconnect-storm conditions that originally exposed bug-1734's gap (this deployment's controller/scheduler connection stayed stable throughout), so the EEXIST/resync branch itself was exercised only via the unit tests above, not a live resync redelivery -- high confidence from code + unit tests, not a live end-to-end repro of the exact original trigger.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1734, bug-1738, bug-1737