--- id: "bug-1815" status: "fixed" status_updated: "2026-09-24" status_note: "" component: "scheduler" severity: "unknown" timestamp: "2026-09-15T09:40:00Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 1 files: ["src/scheduler/src/resource.c (sched_reconcile_host_resource_usage() -- the fix)", "src/scheduler/src/core_events.c (JOBS_PUSH resync-merge RUN branch", "lines ~913-924 -- confirmed the account-object-replacement mechanism that interacts with this gap", "not itself modified)", "src/scheduler/testsuite/test_resource_accounting.c (new regression test)"] tags: ["scheduler", "resource-accounting", "resInUse", "sched_reconcile_host_resource_usage", "IS_WORKER_RECONCILING", "IS_WORKER_UNAVAIL", "worker-rejoin-resync", "bug-1801-gap", "bug-1814-followup", "JOBS_PUSH", "resync", "mem", "never-self-corrects", "user-reported", "live-verified", "found-live", "tdd"] related: ["bug-1814", "bug-1799", "bug-1801", "bug-1751", "bug-1812", "bug-1739"] legacy: true --- # bug-1815: Follow-up to bug-1814: user reported that tracing a job's 'mem used' figure across multiple live… ## Description Follow-up to bug-1814: user reported that tracing a job's 'mem used' figure across multiple live replay sessions showed it never self-corrects, staying elevated until the job finishes -- not a one-time blip. User also live-verified on the real pc13v host (`systemctl show -p MemoryCurrent -p MemoryAccounting`) that systemd genuinely reports correct, sane MemoryCurrent values for a healthy running job (MemoryCurrent=5525504, MemoryAccounting=yes) -- disproving the hypothesis that bug-1814's UINT64_MAX sentinel is a persistent/structural condition on this host. This left the 'why does it never self-correct' question open. ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction _Not recorded (migrated from buglog.json)._ ## Root cause sched_reconcile_host_resource_usage() (the JOB_RUSAGE-driven per-job mem-usage reconciliation path, resource.c) was missing the exact IS_WORKER_UNAVAIL/IS_WORKER_RECONCILING guard that bug-1801 (2026-09-11, same file) already added to sched_update_account() (the regular per-round charge/uncharge path). core_reset_workers() (bug-1751) deliberately FREEZES a RECONCILING/UNAVAIL worker's w->resInUse each round (skips the reset), on the understanding that NOTHING touches it until a clean per-round reset+resum can run once the worker's real state is confirmed; bug-1801 extended that freeze to sched_update_account() but this THIRD, independent resInUse-mutating path was never covered. Consequence: every JOB_RUSAGE tick that lands while a worker is RECONCILING keeps applying its computed delta to resInUse regardless, breaking the freeze. Compounding this: core_events.c's JOBS_PUSH resync-merge RUN branch (lines 913-924) can replace an already-tracked job's entire xce_job_account_t (existing->accounts = job->accounts, after free_job_account()-ing the old one) with a freshly-built object whose 'mem' xce_res_account_t has used=0.0 (make_res_account(), xcealloc.c) -- safe ONLY because the very next clean per-round rebuild (sched_account_jobs(), which fully re-derives resInUse from scratch each round) is supposed to correct any resulting discontinuity. That safety net is exactly what stays skipped while the worker remains RECONCILING. So a rusage tick landing after such a resync-merge, during the same still-RECONCILING window, computes its delta against the FRESH account's 0.0 baseline and adds the FULL usage on top of whatever was already frozen in the pool (potentially including bug-1814's own garbage charge, or simply a prior legitimate charge that now gets double-counted) -- with nothing left to ever subtract it back out until the worker escapes RECONCILING for good (letting a clean rebuild run) or the job reaches a terminal state (whose own uncharge path removes whatever the CURRENT account object holds). This produces exactly the observed 'used' figure that grows and never self-corrects for the job's entire remaining lifetime -- matching the user's live multi-session trace precisely, and consistent with this session's own bug-1812 finding that large array-job bursts cause repeated scheduler-to-controller connection loss (i.e. repeated/extended RECONCILING windows), a plausible trigger for this specific replay's affected jobs. ## Fix FIXED (2026-09-15). Added `if (IS_WORKER_UNAVAIL(w->status) || IS_WORKER_RECONCILING(w->status)) { return 0; }` to sched_reconcile_host_resource_usage() (resource.c), immediately after the existing NULL-arg guard, before any resInUse/acct_entry mutation -- mirroring bug-1801's exact guard on sched_update_account() verbatim (same two status bits, same early-return-as-no-op semantics). Deliberately a full skip (neither the pool NOR acct_entry->used is touched) so the account's own tracked baseline stays consistent with whatever the still-frozen pool holds; once the worker's status clears, the very next rusage tick (or the next clean sched_account_jobs() rebuild) reconciles cleanly with no special-casing needed. TDD: added test_reconcile_host_resource_usage_skips_reconciling_or_unavail_worker (test_resource_accounting.c), mirroring test_charge_account_skips_reconciling_or_unavail_worker_host_charge's (bug-1801) own structure -- asserts a RECONCILING worker's resInUse AND acct_entry->used both stay untouched across a rusage tick, same for UNAVAIL, then confirms reconciliation resumes correctly (overage math verified) on the very next call once status clears. RED confirmed by temporarily disabling the new guard (`if (0 && (...))`) and rebuilding (`xce-build.sh scheduler asan xce.build --incremental`): exactly the 1 new test failed ('Expected 0 Was 5'), 76 other scheduler tests stayed green. GREEN confirmed after restoring the guard: full scheduler suite 77/77 Ok, zero regressions. NOT separately fixed here (out of scope, flagged only): the JOBS_PUSH resync-merge's own free-then-replace-without-uncharge pattern (core_events.c:913-924) remains as-is -- it is safe BY DESIGN as long as the following per-round rebuild is guaranteed to run, which is now consistently true again for all three resInUse-mutating paths (charge/uncharge/reconcile) once this fix closes the last gap; revisiting that pattern itself was judged unnecessary once this symmetry is restored. ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-1814, bug-1799, bug-1801, bug-1751, bug-1812, bug-1739