Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
6.2 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1812 | fixed | 2026-09-24 | scheduler | unknown | 2026-09-15T08:05:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1812: User-reported (interactive sim session=34->35 stepped live, AND independently confirmed via a real…
Description
User-reported (interactive sim session=34->35 stepped live, AND independently confirmed via a real production scheduler.log from rshen@pc13v /t/xce-ha-deploy): job 0-14-1789369200-0[21] holds allocs=1 (state=RUN) at the end of session 34; stepping into session 35 replays a JOBS_PUSH resync (njobs=60, including all of array job 0-14-1789369200-0[0-39]) followed by a JOB_UPDATE and a redundant JOB_STATE_CHANGE(RUN) for [21]; immediately after, info job [21] shows state=RUN but allocs=0. The real scheduler.log shows the identical mechanism live: sched_job_state_change ... state 2 (schedStartJob confirmation) at 15:00:05.196, followed 16ms later by JOBS_PUSH: sched_add_job failed for jobid ...[21] (rc=-17), merging state into existing tracked job instead of discarding -- repeating in bursts on later reconnects (six WARNs in ~2s at 15:01:51-53).
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
CONFIRMED, via direct code reading plus independent live evidence from both a sim replay and a real production scheduler.log. xce_core_push_events()'s JOBS_PUSH handler, on a duplicate jobid (sched_add_job() -EEXIST, a resync redelivery of an already-tracked job), branches three ways once job_state_regresses_terminal() is ruled out: (1) resync claims RUN with real allocation data -> transfer it in (bug-1739's fix); (2) resync claims RUN with NO allocation data -> refuse, keep existing (bug-1739's own companion guard); (3) resync claims anything else (the bare else branch, bug-1734's original fix) -> UNCONDITIONALLY frees existing->accounts/existing->allocs and overwrites existing->state with the resync's own claim, with NO check for whether existing already holds a REAL, already-captured allocation the resync claim doesn't support. on_scheduler_join()'s full resync (xcesched_push_jobs(), src/controller/sched.c:214, fired on every scheduler (re)connect via join.c) is built from collect_jobs_committed() -- a committed-data snapshot gated only on 'job submission is committed', NOT on whether this specific job's DISPATCH transition has been committed yet. When a scheduler reconnects (or the connection blips) around the same moment it makes/receives confirmation of its own fresh dispatch decision for a job, the resync's committed-data snapshot can legitimately still show that job as PEND (its own dispatch commit hasn't propagated into the snapshot query yet) even though the scheduler's own live state (already set via sched_commit_decision() or the schedStartJob/JOB_STATE_CHANGE confirmation path) correctly has it RUN with a real allocation. Branch (3) blindly trusts the resync's stale PEND claim over the scheduler's own more-current, already-allocated state, freeing the real allocation and (in the observed cases) leaving state briefly PEND until a later, in-flight JOB_UPDATE/JOB_STATE_CHANGE for the SAME jobid (already queued in the same event batch, confirming the SAME dispatch) re-sets state=RUN -- but with no mechanism to restore the allocation branch (3) just destroyed. This exactly matches both the sim replay symptom (info jobs shows RUN, info worker's allocs-driven assigned-jobs listing omits it) and the production log's precise 16ms collision. Not the same mechanism as bug-1811's original (correct, but incomplete) 'Recorded-Decision Reconciliation transient window' explanation -- that window is real and self-healing, but this is a genuinely separate, permanent-until-this-fix data-loss bug triggered by resync/dispatch timing, not the reconciliation mechanism itself.
Fix
FIXED (2026-09-15), TDD RED->GREEN. Added a guard in JOBS_PUSH's duplicate-jobid merge (core_events.c), inserted between bug-1739's else if (JOB_IS_RUN(job->state)) branch and the original bare else: else if ((existing->allocs != NULL) && !JOB_IS_DONE(job->state) && !JOB_IS_EXIT(job->state)) -- refuses the resync claim (WARNs, keeps existing->state/allocs/accounts untouched) whenever existing already holds a real allocation AND the resync's claimed state is not genuinely terminal. A terminal claim (DONE/EXIT) still falls through to the original branch, correctly releasing the allocation -- this is bug-1734's own legitimate 'job finished while disconnected' case and must not be blocked. Symmetric in spirit to bug-1739's own phantom-RUN guard (refuse to fabricate an unsupported RUN; now also refuse to destroy a supported RUN). TDD: added test_jobs_push_duplicate_jobid_run_with_allocation_survives_stale_nonrun_resync (test_core_events.c) -- job tracked RUN with a real allocation, redelivered via JOBS_PUSH as PEND with no allocation, must keep its RUN state and real allocation intact; and its companion test_jobs_push_duplicate_jobid_terminal_resync_releases_real_allocation -- the SAME RUN-with-real-allocation job, redelivered as DONE, must still release the allocation and transition (proves the new guard doesn't block legitimate terminal resyncs). RED confirmed (xce-build.sh scheduler asan xce.build --incremental): exactly the new stale-resync test failed (Expected 2 [RUN] Was 1 [PEND]), the terminal-resync companion already passed (unaffected by the unfixed code), 75 other tests green. GREEN confirmed after the fix: full scheduler suite 77/77 Ok, zero regressions -- including test_sim_e2e_original_motivation.c (the Recorded-Decision Reconciliation live-verify a FIRST, incorrect fix attempt this same session had broken and had to be reverted) and test_driver_translate.c, both untouched by this fix and green throughout.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1734, bug-1738, bug-1739, bug-1811