Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
6 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | fix_commits | legacy | gitea_issue | gitea_synced_status | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1799 | fixed | 2026-09-24 | 25fb5c11 |
scheduler | unknown | 2026-09-11T07:32:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
|
true | 268 | fixed |
bug-1799: User-reported (xce_scheduler -i -s , live cluster-captured replay…
Description
User-reported (xce_scheduler -i -s , live cluster-captured replay /tmp/scheduler-replay-20260911-064117.replay, worker pc13v, 22 real slots): info worker pc13v at session=98 shows resource slots: value=22.00 used=22.00 while the printed assigned job: list for that same worker sums to 26 slots (jobs [5]-[9] each *4 = 20, plus six single-slot jobs from a second batch = 6); the very next session (99, no new DISPATCH/PEND events, same assigned-jobs list) shows used=26.00 -- exceeding the worker's own value=22.00 capacity, an admission-invariant violation (max slots exceeded). Session 98's own event stream shows two back-to-back WORKERS_PUSH+JOB_PUSH (njobs=20) events for the same worker/jobids, consistent with a live scheduler-connection resync/reconnect being faithfully replayed (matching bug-1734/bug-1739's documented resync scenario), plus sched_add_job failed ... rc=-17, merging state into existing tracked job instead of discarding warnings for nearly every job in the resync batch, and one resync claims RUN for jobid ...[7]/[8] with no allocation data -- refusing to merge a phantom RUN (bug-1739's guard firing as designed).
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
Interaction between two independently-reasonable pieces of code that were never reconciled with each other. (1) bug-1751 (2026-09-01, worker-rejoin-resync task 5) deliberately added an IS_WORKER_RECONCILING(w->status) skip to core_reset_workers() (core_session.c:136-142, the sim/normal-mode-shared per-round resInUse-reset helper called from xce_core_begin_session()): a worker mid-resync has its w->resInUse zeroing SKIPPED that round, on purpose, so its in-flight usage isn't wiped while reconciliation is still in progress. (2) sched_account_jobs() (scheduler.c:1434-1461) is the ONLY thing that re-derives resInUse/sharedInUse each round -- it walks every job in core->jobs and, for each one with JOB_IS_RUN(job->state) && job->accounts != NULL, calls sched_charge_account() (an unconditional ADD via sched_update_res_accounts(..., charge=1), resource.c:608-633) -- with NO equivalent RECONCILING check and NO reset of its own. Its own documented contract (core.h:311-322) is explicit that callers 'must call this strictly after the session's resval reset' -- a precondition bug-1751's reset-skip silently violates for exactly the RECONCILING window it targets. Consequence: for every round a worker stays RECONCILING (entered via a WORKERS_PUSH re-announce during a scheduler reconnect/resync, exactly what session 98's two WORKERS_PUSH events reflect), every currently-RUN job's slot charge on that worker gets RE-ADDED on top of the NOT-reset resInUse from the prior round, instead of the intended zero-then-resum. This produces exactly the observed shape: used= (accumulating additively) trails one round behind the true assigned jobs sum at first (22 vs 26 at session 98, since only some of the newly-RUN jobs' charges had been double-counted-in by that round's pass), then overtakes real capacity outright (26 at session 99). This is NOT the same mechanism as bug-1739 (JOBS_PUSH resync-merge phantom-RUN guard, which governs whether job->allocs/state get attached at all) nor bug-1624 (already-fixed dispatch-to-RUN synchronization gap) nor bug-1727 (already-fixed info worker display double-count of value=) -- all four live in the same resource-accounting neighborhood but are structurally distinct bugs. Root cause identified via static trace only (codegraph_explore + direct read of core_session.c/scheduler.c/core.h); not yet reproduced with an isolated minimal repro or instrumented trace confirming the exact round-by-round resInUse progression.
Fix
NOT YET FIXED -- investigation only, per user request (this session's live replay-driven Q&A, not yet escalated to an implementation task). Likely fix direction: give sched_account_jobs() the SAME IS_WORKER_RECONCILING awareness core_reset_workers()/sched_reset_worker_() already have (bug-1751) -- either (a) skip charging jobs whose host is currently RECONCILING (mirroring the reset-skip so the pair stays consistent: neither reset nor resum touches a RECONCILING worker's resInUse until it clears), or (b) have core_reset_workers() NOT skip the reset for a RECONCILING worker but instead have sched_account_jobs() re-derive from job state alone regardless of RECONCILING (i.e. resInUse is always a clean per-round sum, and RECONCILING-safety is enforced elsewhere, e.g. admission control refusing new dispatches to a RECONCILING host, which per bug-1751/bug-1745 already exists as a separate candidate-filter check). Needs the worker-rejoin-resync spec's own design intent re-read (.kiro/specs/worker-rejoin-resync/) before picking (a) vs (b), plus a live repro (re-run this exact replay file with an instrumented sched_account_jobs() logging resInUse before/after each round) to confirm the round-by-round mechanism before implementing, and full TDD (scheduler-driver-equivalence steering requires normal-mode/sim-mode parity for any fix here).
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1751, bug-1739, bug-1734, bug-1624, bug-1727, bug-1613
- Gitea: #268