Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
10 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | fix_commits | legacy | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1735 | fixed | 2026-09-24 | d8a37fa2 |
scheduler | unknown | 2026-08-30T10:30:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
|
true |
bug-1735: affects data the scheduler needs for resource-usage-aware scheduling decisions (see priority note…
Description
PRIORITY: HIGH -- affects data the scheduler needs for resource-usage-aware scheduling decisions (see priority note below). Split 2026-08-30 from a broader initial report into this (core usage: memory + CPU time) and bug-1736 (display-only statistics: swap, task count, peak/efficiency percentages) at user request, since the two have very different urgency and the broader one should not block this one.
User-reported: job memory usage always shows 0 in the scheduler's simulation record file (.replay), despite the worker daemon presumably measuring real resource usage. User asked to check whether the worker even reports this data at all -- CONFIRMED: it does, but the value never survives to the controller due to a plain resource-name string mismatch, plus a second, independent gap in how the controller aggregates whatever it does receive.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
TWO CONFIRMED, INDEPENDENT GAPS -- either alone would produce 'memory usage always 0'; both are real. Neither was live-verified this session (no real cgroup-backed xcecd deployment was exercised -- see the environment-limitation note below), but both are confirmed by direct code reading with exact citations, and there is NO translation/mapping layer anywhere between the two mismatched naming conventions (grepped the whole worker/controller/lib/scheduler source tree for both string sets -- only the two sites below reference them at all).
FINDING 1 (the primary, most likely explanation): resource-name string mismatch between producer and consumer. collect_job_rusages() (xcecd_cgroups.c, the REAL, cgroup/systemd-backed worker's rusage collector -- queries a job's own 'xcecd-worker-.service' systemd unit via sd_bus_open_system()) reports exactly 4 resnames: 'cpu_time_sec', 'memory_rss', 'memory_swap', 'tasks_current'. update_host_runusage() (misc.c, the controller-side parser that receives this over the wire) only recognizes the LITERAL strings 'cputime' and 'mem' -- neither of which matches ANY of the 4 names the worker actually sends. Every rusage report from the real worker therefore falls through both if (strcmp(...)) checks doing nothing, silently, for every single field -- host_runusage[j]->cpu_time and ->mem_usage stay at their calloc-zeroed default FOREVER, regardless of what the worker's cgroup query actually measured. No error, no log, no indication anything was dropped -- this is why it reads as 'always 0' rather than an obvious failure.
FINDING 2 (independent, compounding): even if Finding 1 were fixed and host_runusage[]'s PER-HOST fields started populating correctly, the TOP-LEVEL AGGREGATE fields of xce_job_runusage_t (max_mem, avg_mem, used_mem, cpu_time, peak_cpu_time, peak_cpu_duration, mem_efficiency, avg_cpu_efficiency, peak_cpu_efficiency) are NEVER computed from the per-host breakdown anywhere -- update_job_rusages() (job.c) calls update_host_runusage() (which only touches host_runusage[]) then has an explicit, still-open '/* FIXME update other fields of job rusage later */' comment immediately after, with no code implementing that FIXME anywhere in this codebase (grepped). Any downstream consumer that reads the AGGREGATE fields (rather than iterating host_runusage[] itself) would see 0 regardless of Finding 1's fix.
SCHEDULER-SIDE STRUCTURE NOTE (relevant context, not itself a bug): the scheduler does NOT receive or store the aggregated xce_job_runusage_t at all -- xce_sched_job_t.rusages (sched/types.h:115) is typed xce_job_rusages_t*, the SAME flat wire-report struct the worker originally sent (jobid/npids/pids/count/usages[] of raw {resname,value} pairs) -- forwarded/stored as-is, and recording_events.c's capnp encode for the .replay file writes that same raw struct unchanged. So the .replay file the user is inspecting genuinely contains whatever resname/value pairs the worker sent, byte for byte -- if Finding 1 is fixed (or the worker already sent matching names), the RAW data WOULD flow all the way into the recording faithfully. Grepped src/scheduler/src/repl*.c for any resname-aware rendering of memory usage in the interactive REPL/inspect commands and found NONE -- could not determine within this session's time budget whether the specific tool/view the user used to inspect the .replay file (unclear from the report whether it's the live REPL's own display, a separate dump/analysis tool, or something else) has ITS OWN resname expectation that might differ yet again from both 'mem' and 'memory_rss' -- flagged as an open question for whoever picks this up, not confirmed either way.
ENVIRONMENT LIMITATION NOTE: this session's own live burst-testing (bug-1714 investigation) exclusively used xcecd_virtual (the no-root virtual worker), never the real, cgroup/systemd-backed xcecd binary collect_job_rusages() lives in -- this sandbox has no D-Bus/systemd (already confirmed in this session's bug-1731 follow-up work: get_systemd_job_exit_code() always returns -1 here), so sd_bus_open_system() at collect_job_rusages()'s own first real step would itself fail immediately in this environment (returns NULL before ever reaching the resname assignment code) -- this bug could not have been, and was not, live-reproduced in this sandbox; it is a pure static-code finding, applicable specifically to the user's own real deployment (which does have real cgroups/systemd, matching the user's own 'live deployment' framing throughout this investigation).
PRIORITY NOTE (2026-08-30, user request): this half of the original finding -- memory (memory_rss/'mem') and CPU time (cpu_time_sec/'cputime') specifically -- is the HIGH-PRIORITY half. These are exactly the two metrics xce_host_runusage_t already has dedicated fields for (cpu_time, mem_usage) and the two the controller's parser already has matching (if misspelled) if-branches for -- clearly the originally-intended 'core' usage metrics, as opposed to the swap/task-count/peak/efficiency fields split out to bug-1736, which have no controller-side field or computation logic at all today. IMPORTANT CAVEAT, checked directly this session: grepped the full scheduler source tree for any consumer of rusage/usage data in scheduling/picker/allocation logic and found NONE -- xce_sched_job_t.rusages (sched/types.h:115) is stored and forwarded into recordings but not read by any picker plugin or allocation code today. So this fix corrects the DATA PIPELINE (worker measurement -> controller aggregation -> scheduler storage/recording), the necessary prerequisite for any future usage-aware scheduling policy (e.g. backfill sizing, over-commit detection, preemption on real memory pressure) -- it does not itself wire usage data into a scheduling decision, since no such consumer exists yet. Marked high priority on that basis (unblocks correct data for future scheduling work, and is also simply a correctness bug in its own right for any operator-facing usage reporting), not because it changes scheduling behavior today.
Fix
NOT FIXED -- logged for follow-up, scope narrowed 2026-08-30 to memory + CPU time only (see bug-1736 for the split-off lower-priority statistics). RECOMMENDED FIX, Finding 1 (resname mismatch): align 'cpu_time_sec'/'memory_rss' (worker, xcecd_cgroups.c) with 'cputime'/'mem' (controller, misc.c) -- pick a direction (see the two options in the original finding, not repeated here) and apply consistently. RECOMMENDED FIX, Finding 2 (aggregation): implement update_job_rusages()'s (job.c) still-open FIXME specifically for max_mem/avg_mem/used_mem/cpu_time -- derive from host_runusage[]'s per-host entries each time a report merges in (max/avg/sum per field semantics). Both need TDD (unit tests, not just a live check) plus live verification against a REAL cgroup-backed xcecd deployment -- this session's sandbox has no D-Bus/systemd and cannot provide that verification itself (see the original environment-limitation note).
UPDATE (2026-09-03, verification session): CONFIRMED ALREADY FIXED. Both recommended fixes were implemented and committed 2026-08-30 (commit d8a37fa27 for this entry's resname-mismatch Finding 1; commit 5c2c6664e for bug-1736's sibling swap/task-count fields) but this entry was never marked fixed. update_host_runusage() (misc.c) now matches the worker's real resnames ('cpu_time_sec'/'memory_rss'), and aggregate_job_runusage() (misc.c) computes max_mem/avg_mem/used_mem/cpu_time/peak_cpu_time from the per-host breakdown, called from update_job_rusages() (job.c:3502) -- replacing the old FIXME. TDD unit tests exist (src/controller/testsuite/test_host_runusage.c, 7 tests covering resname matching, multi-host update-in-place, swap/task-count parsing, and aggregation single/multi-host/running-max-peak/null-safety). Re-verified this session: fresh /xce-build controller asan, all 6 controller test suites pass including 'host runusage test' (test_host_runusage.c). No code change was needed -- this session's action was verification + closing out the bug-log entry.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1734, bug-1736