Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
9.2 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1814 | fixed | 2026-09-24 | scheduler | unknown | 2026-09-15T09:31:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1814: User-reported, live replay session (/tmp/scheduler-replay-20260915-091847.replay, worker pc13v)…
Description
User-reported, live replay session (/tmp/scheduler-replay-20260915-091847.replay, worker pc13v): info worker pc13v at session=2 (right after a burst of 19 JOB_UPDATE/JOB_RUSAGE event pairs for jobid[0]-[18]) showed resource mem: value=22346629120.00 (20.81 GB) used=334251534843904.000000 (304.00 TB) -- a used figure ~15000x larger than the worker's own total capacity, for a 22-slot/20.81GB host. Session 1 (before any JOB_RUSAGE events) showed a sane used=225280.000000 (220.00 KB) baseline.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
collect_job_rusages() queries 4 systemd cgroup resource counters (CPUUsageNSec, MemoryCurrent, MemorySwapCurrent, TasksCurrent) via get_unit_property()/sd_bus_get_property_trivial(). get_unit_property()'s success return (r>=0) means only that the D-Bus round trip itself worked -- systemd's own convention (undocumented in this codebase before now) is to report UINT64_MAX for a counter that isn't actually being tracked yet for a unit (accounting not yet active, or queried before the unit's cgroup is fully set up), NOT an error and NOT 0. The pre-fix code never checked for this sentinel. For MemoryCurrent specifically: (float)UINT64_MAX / (1024*1024) [MB unit conversion] rounds, under float32, to exactly 17592186044416.0 (2^44) -- a single job's fabricated 'memory usage'. sched_reconcile_job_resource_usage()/sched_reconcile_host_resource_usage() (resource.c, unmodified and correct on its own terms -- resource-accounting.md's max(reserved,actual) policy) then charges this per-job figure into the worker's w->resInUse pool once per RUN job whose rusage tick reports it; the replay session had 19 such jobs report in the same burst, all apparently hitting the same not-yet-tracked-by-systemd condition (freshly-started array-job siblings), so the pool summed to exactly 19 * 2^44 = 334251534843904.0 -- matching the reported figure to the bit. Verified this arithmetic numerically (Python) before making any code change. sched_reconcile_host_resource_usage()'s own reconciliation math (resource.c:802-834) is not itself buggy -- garbage in, garbage charged, exactly as designed for a real (non-sentinel) report.
Fix
FIXED (2026-09-15). Added a new static predicate sd_resource_counter_is_available(uint64_t val) (xcecd_cgroups.c, right after get_unit_property()) that returns false only for the exact UINT64_MAX sentinel, and gated all four collect_job_rusages() property reads (CPUUsageNSec/MemoryCurrent/MemorySwapCurrent/TasksCurrent) on get_unit_property(...) >= 0 && sd_resource_counter_is_available(u64_val) before trusting the value -- an unavailable counter is now silently skipped (the xce_rusage_t entry keeps its calloc-zeroed 0.0 default) rather than treated as a real reading. Applied uniformly to all four properties (not just MemoryCurrent) since systemd uses the identical UINT64_MAX-for-unavailable convention for each of them. Did NOT change memory_rss's existing MB unit convention (bug-1735's established wire contract, also relied on by the controller's own passthrough mem_usage/max_mem/avg_mem/used_mem display pipeline, misc.c/job.c) -- a candidate compensating fix in resource.c's sched_reconcile_job_resource_usage() (multiplying ru->value by 1024*1024 before charging, to match w->resInUse's apparent byte-native convention per resource-accounting.md) was drafted, then reverted: it would have broken test_resource_accounting.c's existing, currently-passing fixtures (test_reconcile_job_resource_usage_maps_memory_rss_to_mem_account and siblings), which assert memory_rss values flow through to w->resInUse with NO unit conversion -- contradicting the codebase's own current tested contract for this function. Left as an OPEN, FLAGGED, NOT-FIXED question (not resolved by this session): whether the scheduler's 'mem' resource job-level reserved/used accounting is actually MB-native end to end (matching the test fixtures and resource-accounting.md's own 'charge 4MB' worked example) or byte-native (matching w->resvals' real mem_free-in-bytes capacity reading) -- these two readings of the same document appear to conflict, and resolving it requires tracing what unit resreq's rusage[mem=...] parsing actually produces in production, which this session did not do. TDD: new src/worker/testsuite/test_xcecd_cgroups_rusage.c (white-box #include of xcecd_cgroups.c, same convention as hardware_monitor_test/resource_sync_test; a trivial make_job_sd_unit_name() stub satisfies the one linker dependency this file pulls in from xcecd_worker_utils.c that this test never actually calls) -- 5 tests covering the sentinel predicate (UINT64_MAX rejected, 0 and ordinary readings accepted, UINT64_MAX-1 accepted i.e. only the exact sentinel is special-cased) plus one test reproducing the exact pre-fix arithmetic (UINT64_MAX/1MB == 17592186044416.0, times 19 == 334251534843904.0) as a documented magnitude proof. The real D-Bus/systemd round trip inside collect_job_rusages() itself could not be live-verified in this sandbox -- no D-Bus/systemd daemon is reachable here (same environment limitation bug-1735 already established for this file); this is a static-analysis-plus-arithmetic-proof fix, needs live confirmation against a real cgroup-backed xcecd deployment. Wired into src/worker/testsuite/meson.build (new 'worker xcecd_cgroups rusage test' target). Full worker suite build+test in progress at time of this entry.
ADDENDUM (2026-09-15, same session, user's own multi-session replay trace): user reported that across multiple replay sessions, a job's corrupted 'used' figure NEVER self-corrects -- it stays elevated until the job finishes, not just at its first rusage tick. This resolves the open 'is this a one-time startup race or a persistent condition' question this entry originally left unanswered: sched_reconcile_host_resource_usage()'s delta-based math (resource.c) means a SUBSEQUENT poll returning a real, small value would compute a large NEGATIVE delta against the still-huge acct_entry->used, self-correcting the pool almost immediately -- that never happening across many sessions proves MemoryCurrent keeps returning the UINT64_MAX sentinel (or something producing an equally-unavailable reading) on EVERY 10-second poll (job_rusage_collect_interval, xcecd_worker.c:50) for that job's entire lifetime, not just once at unit creation. Confirmed MemoryAccounting=yes IS explicitly requested at transient-unit creation (xcecd_cgroups.c:270-291, start_systemd_job()) -- so this is not a 'forgot to enable accounting' bug; something about how MemoryCurrent is actually served back for these units on the live pc13v host (cgroup v1 vs v2, controller delegation, a systemd/D-Bus quirk) appears to be structurally broken, not transiently racy. Separately noted: CPUAccounting is NEVER explicitly requested (only TasksAccounting/MemoryAccounting are) -- CPUUsageNSec's availability depends entirely on that host's system-wide DefaultCPUAccounting= default, a related but distinct potential gap. NOT verified live (no D-Bus/systemd in this sandbox, same limitation as bug-1735) -- needs a check on the real pc13v host (systemctl show <unit> -p MemoryCurrent -p MemoryAccounting and/or cat /sys/fs/cgroup/<cgroup>/memory.current) to distinguish a systemd-layer reporting bug from a kernel/cgroup-delegation-level gap. This fix's guard is still correct and necessary regardless of which it turns out to be -- it stops the garbage charge -- but the practical effect for affected jobs is that 'mem used' will sit at the flat reservation baseline for their whole runtime rather than tracking real usage, since real data genuinely never arrives for them; this is a host/cgroup-configuration question, out of this codebase's scope to resolve further.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1735, bug-1736, bug-1740, bug-1799, bug-1801