Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
38 lines
6.3 KiB
Markdown
38 lines
6.3 KiB
Markdown
---
|
|
id: "bug-1815"
|
|
status: "fixed"
|
|
status_updated: "2026-09-24"
|
|
status_note: ""
|
|
component: "scheduler"
|
|
severity: "unknown"
|
|
timestamp: "2026-09-15T09:40:00Z"
|
|
last_seen: "2026-09-19T09:00:00Z"
|
|
occurrences: 1
|
|
files: ["src/scheduler/src/resource.c (sched_reconcile_host_resource_usage() -- the fix)", "src/scheduler/src/core_events.c (JOBS_PUSH resync-merge RUN branch", "lines ~913-924 -- confirmed the account-object-replacement mechanism that interacts with this gap", "not itself modified)", "src/scheduler/testsuite/test_resource_accounting.c (new regression test)"]
|
|
tags: ["scheduler", "resource-accounting", "resInUse", "sched_reconcile_host_resource_usage", "IS_WORKER_RECONCILING", "IS_WORKER_UNAVAIL", "worker-rejoin-resync", "bug-1801-gap", "bug-1814-followup", "JOBS_PUSH", "resync", "mem", "never-self-corrects", "user-reported", "live-verified", "found-live", "tdd"]
|
|
related: ["bug-1814", "bug-1799", "bug-1801", "bug-1751", "bug-1812", "bug-1739"]
|
|
legacy: true
|
|
---
|
|
|
|
# bug-1815: Follow-up to bug-1814: user reported that tracing a job's 'mem used' figure across multiple live…
|
|
|
|
## Description
|
|
Follow-up to bug-1814: user reported that tracing a job's 'mem used' figure across multiple live replay sessions showed it never self-corrects, staying elevated until the job finishes -- not a one-time blip. User also live-verified on the real pc13v host (`systemctl show <unit> -p MemoryCurrent -p MemoryAccounting`) that systemd genuinely reports correct, sane MemoryCurrent values for a healthy running job (MemoryCurrent=5525504, MemoryAccounting=yes) -- disproving the hypothesis that bug-1814's UINT64_MAX sentinel is a persistent/structural condition on this host. This left the 'why does it never self-correct' question open.
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Root cause
|
|
sched_reconcile_host_resource_usage() (the JOB_RUSAGE-driven per-job mem-usage reconciliation path, resource.c) was missing the exact IS_WORKER_UNAVAIL/IS_WORKER_RECONCILING guard that bug-1801 (2026-09-11, same file) already added to sched_update_account() (the regular per-round charge/uncharge path). core_reset_workers() (bug-1751) deliberately FREEZES a RECONCILING/UNAVAIL worker's w->resInUse each round (skips the reset), on the understanding that NOTHING touches it until a clean per-round reset+resum can run once the worker's real state is confirmed; bug-1801 extended that freeze to sched_update_account() but this THIRD, independent resInUse-mutating path was never covered. Consequence: every JOB_RUSAGE tick that lands while a worker is RECONCILING keeps applying its computed delta to resInUse regardless, breaking the freeze. Compounding this: core_events.c's JOBS_PUSH resync-merge RUN branch (lines 913-924) can replace an already-tracked job's entire xce_job_account_t (existing->accounts = job->accounts, after free_job_account()-ing the old one) with a freshly-built object whose 'mem' xce_res_account_t has used=0.0 (make_res_account(), xcealloc.c) -- safe ONLY because the very next clean per-round rebuild (sched_account_jobs(), which fully re-derives resInUse from scratch each round) is supposed to correct any resulting discontinuity. That safety net is exactly what stays skipped while the worker remains RECONCILING. So a rusage tick landing after such a resync-merge, during the same still-RECONCILING window, computes its delta against the FRESH account's 0.0 baseline and adds the FULL usage on top of whatever was already frozen in the pool (potentially including bug-1814's own garbage charge, or simply a prior legitimate charge that now gets double-counted) -- with nothing left to ever subtract it back out until the worker escapes RECONCILING for good (letting a clean rebuild run) or the job reaches a terminal state (whose own uncharge path removes whatever the CURRENT account object holds). This produces exactly the observed 'used' figure that grows and never self-corrects for the job's entire remaining lifetime -- matching the user's live multi-session trace precisely, and consistent with this session's own bug-1812 finding that large array-job bursts cause repeated scheduler-to-controller connection loss (i.e. repeated/extended RECONCILING windows), a plausible trigger for this specific replay's affected jobs.
|
|
|
|
## Fix
|
|
FIXED (2026-09-15). Added `if (IS_WORKER_UNAVAIL(w->status) || IS_WORKER_RECONCILING(w->status)) { return 0; }` to sched_reconcile_host_resource_usage() (resource.c), immediately after the existing NULL-arg guard, before any resInUse/acct_entry mutation -- mirroring bug-1801's exact guard on sched_update_account() verbatim (same two status bits, same early-return-as-no-op semantics). Deliberately a full skip (neither the pool NOR acct_entry->used is touched) so the account's own tracked baseline stays consistent with whatever the still-frozen pool holds; once the worker's status clears, the very next rusage tick (or the next clean sched_account_jobs() rebuild) reconciles cleanly with no special-casing needed. TDD: added test_reconcile_host_resource_usage_skips_reconciling_or_unavail_worker (test_resource_accounting.c), mirroring test_charge_account_skips_reconciling_or_unavail_worker_host_charge's (bug-1801) own structure -- asserts a RECONCILING worker's resInUse AND acct_entry->used both stay untouched across a rusage tick, same for UNAVAIL, then confirms reconciliation resumes correctly (overage math verified) on the very next call once status clears. RED confirmed by temporarily disabling the new guard (`if (0 && (...))`) and rebuilding (`xce-build.sh scheduler asan xce.build --incremental`): exactly the 1 new test failed ('Expected 0 Was 5'), 76 other scheduler tests stayed green. GREEN confirmed after restoring the guard: full scheduler suite 77/77 Ok, zero regressions. NOT separately fixed here (out of scope, flagged only): the JOBS_PUSH resync-merge's own free-then-replace-without-uncharge pattern (core_events.c:913-924) remains as-is -- it is safe BY DESIGN as long as the following per-round rebuild is guaranteed to run, which is now consistently true again for all three resInUse-mutating paths (charge/uncharge/reconcile) once this fix closes the last gap; revisiting that pattern itself was judged unnecessary once this symmetry is restored.
|
|
|
|
## Verification
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## References
|
|
- Related: bug-1814, bug-1799, bug-1801, bug-1751, bug-1812, bug-1739
|