---
id: "bug-1735"
status: "fixed"
status_updated: "2026-09-24"
status_note: "d8a37fa2"
component: "scheduler"
severity: "unknown"
timestamp: "2026-08-30T10:30:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/worker/src/xcecd_cgroups.c (collect_job_rusages", "line ~1418-1436 -- the worker-side cgroup-based measurement and resname assignment)", "src/controller/misc.c (update_host_runusage", "line ~128-138 -- the controller-side parser)", "src/controller/job.c (update_job_rusages", "line ~3200 -- pre-existing 'FIXME update other fields of job rusage later' comment)", "src/scheduler/include/sched/types.h:115 (xce_sched_job_t.rusages -- confirmed the scheduler stores/forwards the RAW xce_job_rusages_t wire report unmodified", "does not itself re-parse or aggregate it)", "src/scheduler/src/recording_events.c:142 (wire.body.job_rusage = ev->u.job_rusage.rusages -- confirms the .replay file records whatever raw resname/value pairs it received", "unchanged)"]
tags: ["worker", "controller", "resource-usage", "rusage", "memory-tracking", "resname-mismatch", "cgroups", "xcecd_cgroups", "found-via-user-report", "static-analysis-only", "unreproduced-in-sandbox", "no-dbus-systemd-in-sandbox", "needs-real-deployment-verification", "needs-owner", "session-sibling-of-bug-1734", "priority-high", "split-2026-08-30", "core-usage-metrics-only", "verified-2026-09-03"]
related: ["bug-1734", "bug-1736"]
fix_commits: ["d8a37fa2"]
legacy: true
---

# bug-1735: affects data the scheduler needs for resource-usage-aware scheduling decisions (see priority note…

## Description
PRIORITY: HIGH -- affects data the scheduler needs for resource-usage-aware scheduling decisions (see priority note below). Split 2026-08-30 from a broader initial report into this (core usage: memory + CPU time) and bug-1736 (display-only statistics: swap, task count, peak/efficiency percentages) at user request, since the two have very different urgency and the broader one should not block this one.

User-reported: job memory usage always shows 0 in the scheduler's simulation record file (.replay), despite the worker daemon presumably measuring real resource usage. User asked to check whether the worker even reports this data at all -- CONFIRMED: it does, but the value never survives to the controller due to a plain resource-name string mismatch, plus a second, independent gap in how the controller aggregates whatever it does receive.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
TWO CONFIRMED, INDEPENDENT GAPS -- either alone would produce 'memory usage always 0'; both are real. Neither was live-verified this session (no real cgroup-backed xcecd deployment was exercised -- see the environment-limitation note below), but both are confirmed by direct code reading with exact citations, and there is NO translation/mapping layer anywhere between the two mismatched naming conventions (grepped the whole worker/controller/lib/scheduler source tree for both string sets -- only the two sites below reference them at all).

FINDING 1 (the primary, most likely explanation): resource-name string mismatch between producer and consumer. collect_job_rusages() (xcecd_cgroups.c, the REAL, cgroup/systemd-backed worker's rusage collector -- queries a job's own 'xcecd-worker-<jobid>.service' systemd unit via sd_bus_open_system()) reports exactly 4 resnames: 'cpu_time_sec', 'memory_rss', 'memory_swap', 'tasks_current'. update_host_runusage() (misc.c, the controller-side parser that receives this over the wire) only recognizes the LITERAL strings 'cputime' and 'mem' -- neither of which matches ANY of the 4 names the worker actually sends. Every rusage report from the real worker therefore falls through both `if (strcmp(...))` checks doing nothing, silently, for every single field -- host_runusage[j]->cpu_time and ->mem_usage stay at their calloc-zeroed default FOREVER, regardless of what the worker's cgroup query actually measured. No error, no log, no indication anything was dropped -- this is why it reads as 'always 0' rather than an obvious failure.

FINDING 2 (independent, compounding): even if Finding 1 were fixed and host_runusage[]'s PER-HOST fields started populating correctly, the TOP-LEVEL AGGREGATE fields of xce_job_runusage_t (max_mem, avg_mem, used_mem, cpu_time, peak_cpu_time, peak_cpu_duration, mem_efficiency, avg_cpu_efficiency, peak_cpu_efficiency) are NEVER computed from the per-host breakdown anywhere -- update_job_rusages() (job.c) calls update_host_runusage() (which only touches host_runusage[]) then has an explicit, still-open '/* FIXME update other fields of job rusage later */' comment immediately after, with no code implementing that FIXME anywhere in this codebase (grepped). Any downstream consumer that reads the AGGREGATE fields (rather than iterating host_runusage[] itself) would see 0 regardless of Finding 1's fix.

SCHEDULER-SIDE STRUCTURE NOTE (relevant context, not itself a bug): the scheduler does NOT receive or store the aggregated xce_job_runusage_t at all -- xce_sched_job_t.rusages (sched/types.h:115) is typed xce_job_rusages_t*, the SAME flat wire-report struct the worker originally sent (jobid/npids/pids/count/usages[] of raw {resname,value} pairs) -- forwarded/stored as-is, and recording_events.c's capnp encode for the .replay file writes that same raw struct unchanged. So the .replay file the user is inspecting genuinely contains whatever resname/value pairs the worker sent, byte for byte -- if Finding 1 is fixed (or the worker already sent matching names), the RAW data WOULD flow all the way into the recording faithfully. Grepped src/scheduler/src/repl*.c for any resname-aware rendering of memory usage in the interactive REPL/inspect commands and found NONE -- could not determine within this session's time budget whether the specific tool/view the user used to inspect the .replay file (unclear from the report whether it's the live REPL's own display, a separate dump/analysis tool, or something else) has ITS OWN resname expectation that might differ yet again from both 'mem' and 'memory_rss' -- flagged as an open question for whoever picks this up, not confirmed either way.

ENVIRONMENT LIMITATION NOTE: this session's own live burst-testing (bug-1714 investigation) exclusively used xcecd_virtual (the no-root virtual worker), never the real, cgroup/systemd-backed xcecd binary collect_job_rusages() lives in -- this sandbox has no D-Bus/systemd (already confirmed in this session's bug-1731 follow-up work: get_systemd_job_exit_code() always returns -1 here), so sd_bus_open_system() at collect_job_rusages()'s own first real step would itself fail immediately in this environment (returns NULL before ever reaching the resname assignment code) -- this bug could not have been, and was not, live-reproduced in this sandbox; it is a pure static-code finding, applicable specifically to the user's own real deployment (which does have real cgroups/systemd, matching the user's own 'live deployment' framing throughout this investigation).

PRIORITY NOTE (2026-08-30, user request): this half of the original finding -- memory (memory_rss/'mem') and CPU time (cpu_time_sec/'cputime') specifically -- is the HIGH-PRIORITY half. These are exactly the two metrics xce_host_runusage_t already has dedicated fields for (cpu_time, mem_usage) and the two the controller's parser already has matching (if misspelled) if-branches for -- clearly the originally-intended 'core' usage metrics, as opposed to the swap/task-count/peak/efficiency fields split out to bug-1736, which have no controller-side field or computation logic at all today. IMPORTANT CAVEAT, checked directly this session: grepped the full scheduler source tree for any consumer of rusage/usage data in scheduling/picker/allocation logic and found NONE -- xce_sched_job_t.rusages (sched/types.h:115) is stored and forwarded into recordings but not read by any picker plugin or allocation code today. So this fix corrects the DATA PIPELINE (worker measurement -> controller aggregation -> scheduler storage/recording), the necessary prerequisite for any future usage-aware scheduling policy (e.g. backfill sizing, over-commit detection, preemption on real memory pressure) -- it does not itself wire usage data into a scheduling decision, since no such consumer exists yet. Marked high priority on that basis (unblocks correct data for future scheduling work, and is also simply a correctness bug in its own right for any operator-facing usage reporting), not because it changes scheduling behavior today.

## Fix
NOT FIXED -- logged for follow-up, scope narrowed 2026-08-30 to memory + CPU time only (see bug-1736 for the split-off lower-priority statistics). RECOMMENDED FIX, Finding 1 (resname mismatch): align 'cpu_time_sec'/'memory_rss' (worker, xcecd_cgroups.c) with 'cputime'/'mem' (controller, misc.c) -- pick a direction (see the two options in the original finding, not repeated here) and apply consistently. RECOMMENDED FIX, Finding 2 (aggregation): implement update_job_rusages()'s (job.c) still-open FIXME specifically for max_mem/avg_mem/used_mem/cpu_time -- derive from host_runusage[]'s per-host entries each time a report merges in (max/avg/sum per field semantics). Both need TDD (unit tests, not just a live check) plus live verification against a REAL cgroup-backed xcecd deployment -- this session's sandbox has no D-Bus/systemd and cannot provide that verification itself (see the original environment-limitation note).

UPDATE (2026-09-03, verification session): CONFIRMED ALREADY FIXED. Both recommended fixes were implemented and committed 2026-08-30 (commit d8a37fa27 for this entry's resname-mismatch Finding 1; commit 5c2c6664e for bug-1736's sibling swap/task-count fields) but this entry was never marked fixed. update_host_runusage() (misc.c) now matches the worker's real resnames ('cpu_time_sec'/'memory_rss'), and aggregate_job_runusage() (misc.c) computes max_mem/avg_mem/used_mem/cpu_time/peak_cpu_time from the per-host breakdown, called from update_job_rusages() (job.c:3502) -- replacing the old FIXME. TDD unit tests exist (src/controller/testsuite/test_host_runusage.c, 7 tests covering resname matching, multi-host update-in-place, swap/task-count parsing, and aggregation single/multi-host/running-max-peak/null-safety). Re-verified this session: fresh /xce-build controller asan, all 6 controller test suites pass including 'host runusage test' (test_host_runusage.c). No code change was needed -- this session's action was verification + closing out the bug-log entry.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1734, bug-1736