Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
8.8 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1727 | fixed | 2026-09-24 | scheduler | unknown | 2026-08-30T22:00:00Z | 2026-09-19T09:00:00Z | 2 |
|
|
|
true |
bug-1727: User-reported (live REPL, real cluster-captured replay…
Description
User-reported (live REPL, real cluster-captured replay /tmp/scheduler-replay-20260828-085356.replay, 40 sessions, 60 jobs): after xce_scheduler -i -s <file> + step, info worker all shows resource slots: value=44.00 used=22.00 for worker pc13v, but the worker's real installed capacity is 22 slots (confirmed by the same worker's own resource maxcpus: value=22.00 line) and all 22 running jobs are 1-slot-each, so the correct display is value=22.00 used=22.00 (fully packed, 0 free) -- used=22.00 is correct, value=44.00 is double the true installed capacity.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
NOT FIXED -- recorded for future investigation, but well-narrowed: repl_info_worker_one()'s own header comment (repl_info.c:281-291) asserts w->resvals holds 'currently-AVAILABLE' capacity that shrinks as jobs are charged, and that 'value=' should be printed as avail+inUse ('a conservation invariant this codebase's own resource accounting already relies on'), implemented at repl_info.c:376-390's Value_u_fvalue branch: avail_v->u.fvalue + used printed as value=. Traced the actual charge path (sched_account_jobs -> sched_charge_account, resource.c:586 -> sched_update_account, resource.c:559-583 -> sched_update_res_accounts, resource.c:401-470) and confirmed it ONLY ever writes to w->resInUse (sched_update_res_accounts(core, &(w->resInUse), 0, haccount->naccount, haccount->accounts, charge), resource.c:580-581) -- it never touches w->resvals. Exhaustively grepped every zarray_put(&(w->resvals), ...)/zarray_put(&(worker->resvals), ...) call site in the scheduler (core_events.c:500, core_events.c:848, snapshot_load.c:333, schedobj.c:311): all four are WORKERS_PUSH / worker-update / snapshot-restore INGESTION paths only -- i.e. w->resvals is populated purely from what the worker daemon self-reports (its own configured/installed capacity, e.g. 'slots: 22'), and is architecturally NEVER decremented when a job is dispatched or charged. So for a live worker's CONSUMABLE resource like 'slots', w->resvals already IS the total installed capacity (not a live 'remaining free' counter), making repl_info.c's own documented 'avail+used = value, a conservation invariant' assumption appear to be FALSE in practice -- the comment's claimed invariant does not match how w->resvals is actually populated for this resource type. This makes value = avail + used double-count: true installed capacity (22) gets added to itself once resInUse also reads 22, yielding 44. CONFIRMED PRE-EXISTING, NOT caused by today's scheduler-snapshot-job-allocation task 4.6 fix (the sched_account_jobs()-at-load-time change earlier this session): this exact 44/22 pattern already appeared in the user's ORIGINAL first-ever paste this session, produced via the normal step -> sched_account_jobs() path (not the load-time path) -- task 4.6 only changed WHEN the display first goes wrong (now visible immediately at load, previously only after the first step/session, since sched_account_jobs() previously never ran until then). Two competing hypotheses for the real fix, not yet distinguished: (a) repl_info.c's display formula is simply wrong for how resvals is actually populated -- should print avail_v directly as value= (already the total) with used= from resInUse, no addition; or (b) w->resvals was INTENDED to be a live 'remaining free' counter (matching the comment's claimed invariant) and something is missing a w->resvals -= charged_amount step at dispatch/charge time that was never implemented -- but no evidence of (b) was found anywhere in resource.c's charge/uncharge functions, so (a) is the better-supported hypothesis pending closer investigation of what a live worker daemon (xcecd) actually reports for a 'slots' resval on each heartbeat (constant install size vs. something already load-adjusted).
--- CONFIRMED + FIXED 2026-08-30 (scheduler-framework session, user's live Symptom 2 report) ---
Hypothesis (a) from the prior investigation is CONFIRMED correct, and hypothesis (b) is definitively ruled out. Live-reproduced end to end: built a fresh scheduler+controller+worker (schema change for a separate bug bundled in the same build, unrelated to this fix), deployed a real 3-node HA cluster + xcecd_virtual worker (22 real cores, matching the user's original pc13v report), submitted a 50-element array job (sleep 60 && hostname, overlapping runtime) against the 22-slot worker, captured a scheduler recording mid-run (step 100, 19 jobs genuinely RUN), and inspected info worker worker-1 on the UNPATCHED binary first: resource slots: value=41.00 used=19.00 -- NOT even a clean 2x (44), because avail_v (=22, the worker's real installed capacity, confirmed never decremented -- grepped every zarray_put(&w->resvals,...) call site: all four (core_events.c:500/848, snapshot_load.c:333, schedobj.c:311) are WORKERS_PUSH/RESVALS_DELTA/snapshot-restore INGESTION only) plus used_v (=19, however many jobs happen to be running at inspection time) are ADDED together by repl_info_worker_one()'s own display formula -- so the displayed 'value' grows in lockstep with 'used' as more jobs run, not a fixed doubling. The ORIGINAL user report's exact 22+22=44 was simply the specific case of a fully-packed worker (used == the worker's own real capacity), which is why it read as a clean 2x. Cross-checked against the actual admission-control formula that this exact w->resvals field feeds -- resource.c's unsatisfy_(): total[i] < inuse[i] + uncomitted[i] + wanted[i] -- which treats get_resvals(&w->resvals,...) as total directly, already inclusive of whatever is inuse. This is airtight proof w->resvals[idx] is the STATIC TOTAL, not a live 'remaining available' counter that would need used added back to reconstruct the total (hypothesis (b), now ruled out -- no such 'w->resvals -= charged_amount' step exists anywhere, and none should, since the admission check already subtracts inuse separately). The real scheduling/admission decisions are UNAFFECTED by this bug -- they read w->resvals and w->resInUse directly, not through this display formula -- so job placement correctness was never at risk; this is a pure info worker/info resource REPL display defect. IMPORTANT DISAMBIGUATION: this display bug is entirely SEPARATE from Symptom 1 (phantom RUN jobs exceeding real worker capacity, logged as bug-1739) -- that symptom is a genuine allocation-bookkeeping gap in bug-1734's JOBS_PUSH resync-merge path, unrelated to this printf-time double-count.
Rebuilt with the SAME binary against the SAME live-captured replay file: resource slots: value=22.00 used=19.00 -- matches resource maxcpus: value=22.00 exactly, confirming the fix.
Fix
FIXED (2026-08-30). src/scheduler/src/repl_info.c: repl_info_worker_one()'s per-worker CONSUMABLE-resource branch and repl_info_resource_one()'s identical GLOBAL/SHARED-resource branch both changed from printing avail + used as value= to printing avail alone (used= is unchanged, still read from w->resInUse / core->sharedInUse) -- avail_v already IS the total capacity, so adding used on top double-counted it. Corrected the header comment above repl_info_worker_one() (previously asserted a false 'avail+used is a conservation invariant' claim) to document the real semantics, with a citation to resource.c's unsatisfy_() as the proof. TDD: updated the 2 existing tests that had baked in the wrong expected value (test_repl_breakpoints_inspection.c: test_info_worker_reports_consumable_value_and_used, expected string changed from 'value=7.00 used=2.00' to 'value=5.00 used=2.00' for avail=5/used=2; test_info_resource_reports_consumable_value_and_used, 'value=11.00 used=3.00' to 'value=8.00 used=3.00' for avail=8/used=3) -- both now correctly assert the raw total, not the double-counted sum. The two tests in test_sim_e2e_original_motivation.c asserting 'value=32.00 used=0.00' / 'value=15.00 used=0.00' needed no change (used=0 makes the old and new formulas identical). Full scheduler + controller suite: 61/61 Ok after the fix, zero regressions. Live re-verified against the real captured replay: value now matches maxcpus exactly at every inspection point (mid-run and post-completion).
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1568, bug-1590