Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
6.3 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1815 | fixed | 2026-09-24 | scheduler | unknown | 2026-09-15T09:40:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1815: Follow-up to bug-1814: user reported that tracing a job's 'mem used' figure across multiple live…
Description
Follow-up to bug-1814: user reported that tracing a job's 'mem used' figure across multiple live replay sessions showed it never self-corrects, staying elevated until the job finishes -- not a one-time blip. User also live-verified on the real pc13v host (systemctl show <unit> -p MemoryCurrent -p MemoryAccounting) that systemd genuinely reports correct, sane MemoryCurrent values for a healthy running job (MemoryCurrent=5525504, MemoryAccounting=yes) -- disproving the hypothesis that bug-1814's UINT64_MAX sentinel is a persistent/structural condition on this host. This left the 'why does it never self-correct' question open.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
sched_reconcile_host_resource_usage() (the JOB_RUSAGE-driven per-job mem-usage reconciliation path, resource.c) was missing the exact IS_WORKER_UNAVAIL/IS_WORKER_RECONCILING guard that bug-1801 (2026-09-11, same file) already added to sched_update_account() (the regular per-round charge/uncharge path). core_reset_workers() (bug-1751) deliberately FREEZES a RECONCILING/UNAVAIL worker's w->resInUse each round (skips the reset), on the understanding that NOTHING touches it until a clean per-round reset+resum can run once the worker's real state is confirmed; bug-1801 extended that freeze to sched_update_account() but this THIRD, independent resInUse-mutating path was never covered. Consequence: every JOB_RUSAGE tick that lands while a worker is RECONCILING keeps applying its computed delta to resInUse regardless, breaking the freeze. Compounding this: core_events.c's JOBS_PUSH resync-merge RUN branch (lines 913-924) can replace an already-tracked job's entire xce_job_account_t (existing->accounts = job->accounts, after free_job_account()-ing the old one) with a freshly-built object whose 'mem' xce_res_account_t has used=0.0 (make_res_account(), xcealloc.c) -- safe ONLY because the very next clean per-round rebuild (sched_account_jobs(), which fully re-derives resInUse from scratch each round) is supposed to correct any resulting discontinuity. That safety net is exactly what stays skipped while the worker remains RECONCILING. So a rusage tick landing after such a resync-merge, during the same still-RECONCILING window, computes its delta against the FRESH account's 0.0 baseline and adds the FULL usage on top of whatever was already frozen in the pool (potentially including bug-1814's own garbage charge, or simply a prior legitimate charge that now gets double-counted) -- with nothing left to ever subtract it back out until the worker escapes RECONCILING for good (letting a clean rebuild run) or the job reaches a terminal state (whose own uncharge path removes whatever the CURRENT account object holds). This produces exactly the observed 'used' figure that grows and never self-corrects for the job's entire remaining lifetime -- matching the user's live multi-session trace precisely, and consistent with this session's own bug-1812 finding that large array-job bursts cause repeated scheduler-to-controller connection loss (i.e. repeated/extended RECONCILING windows), a plausible trigger for this specific replay's affected jobs.
Fix
FIXED (2026-09-15). Added if (IS_WORKER_UNAVAIL(w->status) || IS_WORKER_RECONCILING(w->status)) { return 0; } to sched_reconcile_host_resource_usage() (resource.c), immediately after the existing NULL-arg guard, before any resInUse/acct_entry mutation -- mirroring bug-1801's exact guard on sched_update_account() verbatim (same two status bits, same early-return-as-no-op semantics). Deliberately a full skip (neither the pool NOR acct_entry->used is touched) so the account's own tracked baseline stays consistent with whatever the still-frozen pool holds; once the worker's status clears, the very next rusage tick (or the next clean sched_account_jobs() rebuild) reconciles cleanly with no special-casing needed. TDD: added test_reconcile_host_resource_usage_skips_reconciling_or_unavail_worker (test_resource_accounting.c), mirroring test_charge_account_skips_reconciling_or_unavail_worker_host_charge's (bug-1801) own structure -- asserts a RECONCILING worker's resInUse AND acct_entry->used both stay untouched across a rusage tick, same for UNAVAIL, then confirms reconciliation resumes correctly (overage math verified) on the very next call once status clears. RED confirmed by temporarily disabling the new guard (if (0 && (...))) and rebuilding (xce-build.sh scheduler asan xce.build --incremental): exactly the 1 new test failed ('Expected 0 Was 5'), 76 other scheduler tests stayed green. GREEN confirmed after restoring the guard: full scheduler suite 77/77 Ok, zero regressions. NOT separately fixed here (out of scope, flagged only): the JOBS_PUSH resync-merge's own free-then-replace-without-uncharge pattern (core_events.c:913-924) remains as-is -- it is safe BY DESIGN as long as the following per-round rebuild is guaranteed to run, which is now consistently true again for all three resInUse-mutating paths (charge/uncharge/reconcile) once this fix closes the last gap; revisiting that pattern itself was judged unnecessary once this symmetry is restored.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1814, bug-1799, bug-1801, bug-1751, bug-1812, bug-1739