myxce/.wolf/bugs/bug-1736.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	legacy	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1736	fixed	2026-10-02		controller	unknown	2026-08-30T10:45:00Z	2026-09-19T09:00:00Z	1	src/worker/src/xcecd_cgroups.c (collect_job_rusages -- already sends memory_swap/tasks_current no controller-side field to receive them) src/include/xce/types.h (xce_host_runusage_t -- would need new fields for swap/task-count; xce_job_runusage_t -- the peak/efficiency fields with no producer) src/controller/misc.c (update_host_runusage -- would need new parse branches) src/controller/job.c (update_job_rusages -- same open FIXME as bug-1735 this is the other half of what it would need to cover)	worker controller resource-usage rusage display-only reporting-only priority-low split-2026-08-30 no-scheduling-impact static-analysis-only unreproduced-in-sandbox needs-design-not-just-fix peak-cpu-time-fixed swap-taskcount-fixed efficiency-fields-remain-scope-decision	bug-1735	5c2c6664 6df644e6 1cedd7f7	true	344	fixed	50ebc3869377

bug-1736: job CPU peak and efficiency usage fields are never computed

Description

PRIORITY: LOW -- display/reporting only, not consumed by any scheduling logic. Split 2026-08-30 from bug-1735 at user request: the 'statistics-only' half of that finding -- resource usage fields with no scheduling relevance, purely for operator-facing job accounting/reporting (bjobs -l/bhist-style views), as opposed to bug-1735's memory+CPU-time core, which is a prerequisite for future scheduling decisions. Worker (collect_job_rusages(), xcecd_cgroups.c) already measures and sends 'memory_swap' and 'tasks_current' per job, but the controller has NO field to receive them at all (xce_host_runusage_t only has cpu_time/mem_usage -- no swap or task-count field exists) -- these two are silently dropped entirely, not just misnamed. Separately, xce_job_runusage_t's peak/efficiency fields (peak_cpu_time, peak_cpu_duration, mem_efficiency, avg_cpu_efficiency, peak_cpu_efficiency) have no computation logic anywhere in the codebase (grepped) -- nothing populates them, ever, regardless of bug-1735's fix.

Environment

Not recorded (migrated from buglog.json).

Reproduction

Not recorded (migrated from buglog.json).

Root cause

See bug-1735's root_cause for the full original investigation (resname-mismatch mechanism, environment-limitation note re: this session's sandbox having no real cgroup-backed worker to live-verify against) -- this entry covers only the subset of that finding split out as lower-priority. Unlike bug-1735's memory/cpu_time (which have existing-but-mismatched fields), memory_swap/tasks_current have NO receiving field in xce_host_runusage_t at all -- the worker's measurement is a complete dead end on the controller side, not a naming mismatch. The peak/efficiency fields (xce_job_runusage_t's own top-level fields) are simply never computed by anything, anywhere -- confirmed via grep, no partial implementation exists to build on.

Fix

(1) memory_swap/tasks_current and (2) peak_cpu_time-as-running-max -- 2026-08-30, 5c2c6664 (see git history).

(3) 6df644e6, 2026-10-01: the remaining fields, with LSF bjobs -l definitions (chosen by the user). update_job_cpu_stats() (src/controller/misc.c), called by update_job_rusages() (job.c) after each report:

• each host's cpu_rate = CPU-seconds gained / wall seconds since that host's previous report (sample_time); the job's cores = sum over its hosts. A report no later than the previous one, or with a CPU counter that went backwards, leaves the rate alone.
• peak_cpu = highest job cores seen; peak_cpu_time = it rounded (the wire field is Int32); peak_cpu_duration = the interval that set it. This replaces the earlier peak_cpu_time = running max of cumulative CPU seconds, which only duplicated cpu_time.
• avg_cpu_efficiency = cpu_time / elapsed since start / allocated slots; peak_cpu_efficiency = peak_cpu / slots; mem_efficiency = highest used_mem / the job-wide rusage[mem] request (calc_wanted_job_total_resval, per task by default, as the scheduler reserves it); all in percent, as the apiserver/xcecli models document; 0 when there is no slot count or no memory request (or the request is not a single primary one). New persisted fields HostRunusage.sampleTime/cpuRate and JobRunusage.peakCpu are appended to controller.capnp (old WAL entries still decode) and copied by merge_job_runusage(). The values are controller-side only: the apiserver does not map run usage into its job model and xcecli/harness do not display these fields.

UPDATE (2026-10-05): Exposed to the apiserver and xcecli (2026-10-05): the capnp Job struct gains runUsage @13 (JobRunusage, mapped to xce_job_t.run_usage), so the job pushed to the apiserver on every update (update_job_rusages fires HOOK_DETAIL_JOB_UPDATE after recomputing the stats) carries the values. The same struct is the snapshot/WAL job format, so checkpoints now keep run_usage too (before, a job restored from a snapshot lost it: the WAL_JOB_RUSAGE facts behind the checkpoint are trimmed). Old entries decode with no run usage. apiserver: convert.go/convert_pure.go map run_usage into JobInfo, which gains max_mem/avg_mem/used_mem/peak_cpu/peak_cpu_time/peak_cpu_duration/mem_efficiency/avg_cpu_efficiency/peak_cpu_efficiency columns (moved out of the validation-only JobAdditionalFields; cpu_time was there but never set). xcecli: new -o fields MAX_MEM, AVG_MEM, MEM_EFFICIENCY, CPU_PEAK, CPU_PEAK_DURATION, CPU_EFFICIENCY, CPU_PEAK_EFFICIENCY (CPU_TIME now renders too; all printed '-' before), and jobs list -l prints an LSF bjobs -l style MEMORY USAGE / CPU USAGE block for started jobs.

Verification

• Test: test_cpu_stats_* (rate/peak/duration/efficiency, multi-host sum, mem efficiency as a peak, guards) and test_aggregate_job_runusage_leaves_peak_cpu_alone in src/controller/testsuite/test_host_runusage.c (11/11; written before update_job_cpu_stats() existed); WAL round trip of the new fields added to test_wal_job_rusage_fact.c, and the existing-host merge test test_wru_apply_committed_entry_merges_without_freeing_live_host: RED with the new merge_job_runusage() lines removed (Expected 3.25 Was 0), GREEN with them
• Suites: controller_unit_test 1275/1275; xce-build.sh all asan build.jai 126/127 -- the one failure was the known intermittent harness pty timeout bug-1902 (recurrence recorded; 3/3 reruns pass)
• Live: n/a (no consumer displays these fields; the computation is unit-tested against the same report sequence update_job_rusages() applies)

UPDATE (2026-10-05): Exposure (2026-10-05): test_snapent_checkpoint_run_usage_roundtrip (test_persist_snapshot_entities.c, every job- and host-level field through checkpoint frames and persist_load_snapshot; RED before: run_usage NULL), TestRawJobToModel_RunUsage / NoRunUsage / TestJobInfoExposesRunUsageColumns (apiserver backend), TestJobResourceUsageLines* / TestJobRunUsageColumn / TestJobFieldsMapIncludesRunUsage (xcecli output), all RED before. Field-completeness check (audit dimension 7, run_usage only): encoded by capnp_encode_job (snapshot) and WAL_JOB_RUSAGE; adopted by restore_job/job_init (MOVE_PTR) and both apply seams via merge_job_runusage. xce-build all asan: 127/127 + Go ok. Live: single controller + scheduler + apiserver (run-resource-verify.sh --keep-running), an xce_test fake worker reporting cpu_time_sec=100 / memory_rss=200 every 60 s: xcecli jobs list -o showed CPU_TIME 100, MAX_MEM 200, AVG_MEM 200, CPU_EFFICIENCY 50.00% (100 s over ~200 s elapsed, 1 slot), CPU_PEAK 0.00 (the fake counter is constant, so the rate is 0), MEM_EFFICIENCY 0.00% (no rusage[mem] request); jobs list -l printed the same in the MEMORY USAGE / CPU USAGE block.

UPDATE (2026-10-05): Format change under HA (2026-10-05): /xce-test-ha sequential 20 rounds PASS (zero job loss, all 3 nodes led, no ASAN, WAL segment layout clean); its virtual worker sends no usage reports, so those jobs carry a null runUsage -- the populated round trip is covered by test_snapent_checkpoint_run_usage_roundtrip.

References

• Related: bug-1735