docs(buglog): bug-1736 fixed (6df644e6); bug-1902 recurred
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
6df644e6d5
commit
6eaccc7476
GPG key ID:
80533536B0EB651C
3 changed files with 31 additions and 30 deletions
|
|
@ -11425,18 +11425,18 @@
|
|||
},
|
||||
{
|
||||
"id": "bug-1736",
|
||||
"status": "partial",
|
||||
"status_updated": "2026-09-24",
|
||||
"status_note": "swap/tasks/peak_cpu_time fixed 5c2c6664; peak_cpu_duration/efficiency still not computed",
|
||||
"status": "fixed",
|
||||
"status_updated": "2026-10-02",
|
||||
"status_note": "",
|
||||
"component": "controller",
|
||||
"severity": "unknown",
|
||||
"timestamp": "2026-08-30T10:45:00Z",
|
||||
"last_seen": "2026-09-19T09:00:00Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "display/reporting only, not consumed by any scheduling logic\n\nPRIORITY: LOW -- display/reporting only, not consumed by any scheduling logic. Split 2026-08-30 from bug-1735 at user request: the 'statistics-only' half of that finding -- resource usage fields with no scheduling relevance, purely for operator-facing job accounting/reporting (bjobs -l/bhist-style views), as opposed to bug-1735's memory+CPU-time core, which is a prerequisite for future scheduling decisions. Worker (collect_job_rusages(), xcecd_cgroups.c) already measures and sends 'memory_swap' and 'tasks_current' per job, but the controller has NO field to receive them at all (xce_host_runusage_t only has cpu_time/mem_usage -- no swap or task-count field exists) -- these two are silently dropped entirely, not just misnamed. Separately, xce_job_runusage_t's peak/efficiency fields (peak_cpu_time, peak_cpu_duration, mem_efficiency, avg_cpu_efficiency, peak_cpu_efficiency) have no computation logic anywhere in the codebase (grepped) -- nothing populates them, ever, regardless of bug-1735's fix.",
|
||||
"error_message": "job CPU peak and efficiency usage fields are never computed\n\nPRIORITY: LOW -- display/reporting only, not consumed by any scheduling logic. Split 2026-08-30 from bug-1735 at user request: the 'statistics-only' half of that finding -- resource usage fields with no scheduling relevance, purely for operator-facing job accounting/reporting (bjobs -l/bhist-style views), as opposed to bug-1735's memory+CPU-time core, which is a prerequisite for future scheduling decisions. Worker (collect_job_rusages(), xcecd_cgroups.c) already measures and sends 'memory_swap' and 'tasks_current' per job, but the controller has NO field to receive them at all (xce_host_runusage_t only has cpu_time/mem_usage -- no swap or task-count field exists) -- these two are silently dropped entirely, not just misnamed. Separately, xce_job_runusage_t's peak/efficiency fields (peak_cpu_time, peak_cpu_duration, mem_efficiency, avg_cpu_efficiency, peak_cpu_efficiency) have no computation logic anywhere in the codebase (grepped) -- nothing populates them, ever, regardless of bug-1735's fix.",
|
||||
"file": "src/worker/src/xcecd_cgroups.c (collect_job_rusages -- already sends memory_swap/tasks_current, no controller-side field to receive them), src/include/xce/types.h (xce_host_runusage_t -- would need new fields for swap/task-count; xce_job_runusage_t -- the peak/efficiency fields with no producer), src/controller/misc.c (update_host_runusage -- would need new parse branches), src/controller/job.c (update_job_rusages -- same open FIXME as bug-1735, this is the other half of what it would need to cover)",
|
||||
"root_cause": "See bug-1735's root_cause for the full original investigation (resname-mismatch mechanism, environment-limitation note re: this session's sandbox having no real cgroup-backed worker to live-verify against) -- this entry covers only the subset of that finding split out as lower-priority. Unlike bug-1735's memory/cpu_time (which have existing-but-mismatched fields), memory_swap/tasks_current have NO receiving field in xce_host_runusage_t at all -- the worker's measurement is a complete dead end on the controller side, not a naming mismatch. The peak/efficiency fields (xce_job_runusage_t's own top-level fields) are simply never computed by anything, anywhere -- confirmed via grep, no partial implementation exists to build on.",
|
||||
"fix": "PARTIALLY FIXED (2026-08-30). Of the two gaps this entry covers:\n\n(1) memory_swap/tasks_current -- FIXED. Added mem_swap (double)/tasks_current (int) fields to xce_host_runusage_t (types.h), matching capnp fields (memSwap @3/tasksCurrent @4) appended to the HostRunusage schema (controller.capnp) for WAL field-completeness, parsing in update_host_runusage() (misc.c) for the worker's 'memory_swap'/'tasks_current' resnames, and per-host copy in merge_job_runusage()'s (job.c) existing-host branch (the new fields would otherwise have been silently dropped on WAL replay/catchup for a host already seen).\n\n(2) peak_cpu_time -- FIXED. aggregate_job_runusage() (misc.c) now tracks it as the running MAX of the per-call cpu_time sum across every report a job has ever received (not just the latest), guarding against a later report reducing it (e.g. an out-of-order merge/replay).\n\nSTILL NOT FIXED, DELIBERATELY (documented scope decision, not an oversight -- see aggregate_job_runusage()'s own doc comment in misc.c): peak_cpu_duration and the three efficiency fields (mem_efficiency/avg_cpu_efficiency/peak_cpu_efficiency). These need a 'used vs. requested' ratio against a job's resource request (a resreq expression tree, not a plain core/slot count) and/or an unspecified duration semantic -- genuine product/design decisions, not a formula that can be safely guessed. Left at their existing zero value rather than populated with an invented number that could mislead an operator.\n\nVerified: full `meson test --suite controller` clean, Ok:6 Fail:0, 1220/1220 individual controller_unit_test cases pass (zero regression from touching the shared xce_host_runusage_t struct + its capnp schema), plus 2 new regression tests in test_host_runusage.c (7/7 cases pass).",
|
||||
"fix": "(1) memory_swap/tasks_current and (2) peak_cpu_time-as-running-max -- 2026-08-30, 5c2c6664 (see git history).\n\n(3) 6df644e6, 2026-10-01: the remaining fields, with LSF bjobs -l definitions (chosen by the user). `update_job_cpu_stats()` (`src/controller/misc.c`), called by `update_job_rusages()` (`job.c`) after each report:\n- each host's `cpu_rate` = CPU-seconds gained / wall seconds since that host's previous report (`sample_time`); the job's cores = sum over its hosts. A report no later than the previous one, or with a CPU counter that went backwards, leaves the rate alone.\n- `peak_cpu` = highest job cores seen; `peak_cpu_time` = it rounded (the wire field is Int32); `peak_cpu_duration` = the interval that set it. This replaces the earlier `peak_cpu_time` = running max of cumulative CPU seconds, which only duplicated `cpu_time`.\n- `avg_cpu_efficiency` = cpu_time / elapsed since start / allocated slots; `peak_cpu_efficiency` = peak_cpu / slots; `mem_efficiency` = highest used_mem / the job-wide `rusage[mem]` request (`calc_wanted_job_total_resval`, per task by default, as the scheduler reserves it); all in percent, as the apiserver/xcecli models document; 0 when there is no slot count or no memory request (or the request is not a single primary one).\nNew persisted fields `HostRunusage.sampleTime`/`cpuRate` and `JobRunusage.peakCpu` are appended to `controller.capnp` (old WAL entries still decode) and copied by `merge_job_runusage()`. The values are controller-side only: the apiserver does not map run usage into its job model and xcecli/harness do not display these fields.\n\nVerification:\n- Test: `test_cpu_stats_*` (rate/peak/duration/efficiency, multi-host sum, mem efficiency as a peak, guards) and `test_aggregate_job_runusage_leaves_peak_cpu_alone` in `src/controller/testsuite/test_host_runusage.c` (11/11; written before `update_job_cpu_stats()` existed); WAL round trip of the new fields added to `test_wal_job_rusage_fact.c`, and the existing-host merge test `test_wru_apply_committed_entry_merges_without_freeing_live_host`: RED with the new `merge_job_runusage()` lines removed (`Expected 3.25 Was 0`), GREEN with them\n- Suites: controller_unit_test 1275/1275; `xce-build.sh all asan build.jai` 126/127 -- the one failure was the known intermittent harness pty timeout bug-1902 (recurrence recorded; 3/3 reruns pass)\n- Live: n/a (no consumer displays these fields; the computation is unit-tested against the same report sequence update_job_rusages() applies)",
|
||||
"tags": [
|
||||
"worker",
|
||||
"controller",
|
||||
|
|
@ -11458,7 +11458,8 @@
|
|||
"bug-1735"
|
||||
],
|
||||
"fix_commits": [
|
||||
"5c2c6664"
|
||||
"5c2c6664",
|
||||
"6df644e6"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1736.md"
|
||||
},
|
||||
|
|
@ -16818,14 +16819,14 @@
|
|||
},
|
||||
{
|
||||
"id": "bug-1902",
|
||||
"status": "cannot-reproduce",
|
||||
"status_updated": "2026-10-01",
|
||||
"status_note": "seen once during task 4.3; passed 5 subsequent runs",
|
||||
"status": "investigating",
|
||||
"status_updated": "2026-10-02",
|
||||
"status_note": "recurred 2026-10-01 in a full xce-build all run (bug-1736 work): test_walcompare_unrecognized_flag_rejected_with_usage, 'xce_test did not exit within the harness's safety timeout after quit'; passed 3/3 targeted reruns",
|
||||
"component": "harness",
|
||||
"severity": "noise",
|
||||
"timestamp": "2026-10-01T11:22:43Z",
|
||||
"last_seen": "2026-10-01T11:22:43Z",
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-10-02T04:14:22Z",
|
||||
"occurrences": 2,
|
||||
"error_message": "harness pty test test_source_missing_file_rejected_with_error timed out once after quit\n\n**Observed:** during task 4.3 of spec job-array-one-based-index, one run of the harness suite under ASAN failed in the pty CLI test:\n```\ntest_test_cli.c: test_source_missing_file_rejected_with_error: FAIL: xce_test did not exit within the harness's safety timeout after 'quit'\n```\n**Expected:** after `quit`, `xce_test` exits within the test's safety timeout and the test passes, as it does in every other run.\n**Impact:** noise: an intermittent false failure of the harness suite; no product impact observed.\n**Found by:** test (harness suite run during task 4.3, 2026-10-01)",
|
||||
"file": "src/harness/testsuite/test_test_cli.c",
|
||||
"root_cause": "TODO: required for fixed/partial",
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
---
|
||||
id: "bug-1736"
|
||||
status: "partial"
|
||||
status_updated: "2026-09-24"
|
||||
status_note: "swap/tasks/peak_cpu_time fixed 5c2c6664; peak_cpu_duration/efficiency still not computed"
|
||||
status: "fixed"
|
||||
status_updated: "2026-10-02"
|
||||
status_note: ""
|
||||
component: "controller"
|
||||
severity: "unknown"
|
||||
timestamp: "2026-08-30T10:45:00Z"
|
||||
|
|
@ -11,11 +11,11 @@ occurrences: 1
|
|||
files: ["src/worker/src/xcecd_cgroups.c (collect_job_rusages -- already sends memory_swap/tasks_current", "no controller-side field to receive them)", "src/include/xce/types.h (xce_host_runusage_t -- would need new fields for swap/task-count; xce_job_runusage_t -- the peak/efficiency fields with no producer)", "src/controller/misc.c (update_host_runusage -- would need new parse branches)", "src/controller/job.c (update_job_rusages -- same open FIXME as bug-1735", "this is the other half of what it would need to cover)"]
|
||||
tags: ["worker", "controller", "resource-usage", "rusage", "display-only", "reporting-only", "priority-low", "split-2026-08-30", "no-scheduling-impact", "static-analysis-only", "unreproduced-in-sandbox", "needs-design-not-just-fix", "peak-cpu-time-fixed", "swap-taskcount-fixed", "efficiency-fields-remain-scope-decision"]
|
||||
related: ["bug-1735"]
|
||||
fix_commits: ["5c2c6664"]
|
||||
fix_commits: ["5c2c6664", "6df644e6"]
|
||||
legacy: true
|
||||
---
|
||||
|
||||
# bug-1736: display/reporting only, not consumed by any scheduling logic
|
||||
# bug-1736: job CPU peak and efficiency usage fields are never computed
|
||||
|
||||
## Description
|
||||
PRIORITY: LOW -- display/reporting only, not consumed by any scheduling logic. Split 2026-08-30 from bug-1735 at user request: the 'statistics-only' half of that finding -- resource usage fields with no scheduling relevance, purely for operator-facing job accounting/reporting (bjobs -l/bhist-style views), as opposed to bug-1735's memory+CPU-time core, which is a prerequisite for future scheduling decisions. Worker (collect_job_rusages(), xcecd_cgroups.c) already measures and sends 'memory_swap' and 'tasks_current' per job, but the controller has NO field to receive them at all (xce_host_runusage_t only has cpu_time/mem_usage -- no swap or task-count field exists) -- these two are silently dropped entirely, not just misnamed. Separately, xce_job_runusage_t's peak/efficiency fields (peak_cpu_time, peak_cpu_duration, mem_efficiency, avg_cpu_efficiency, peak_cpu_efficiency) have no computation logic anywhere in the codebase (grepped) -- nothing populates them, ever, regardless of bug-1735's fix.
|
||||
|
|
@ -30,18 +30,18 @@ _Not recorded (migrated from buglog.json)._
|
|||
See bug-1735's root_cause for the full original investigation (resname-mismatch mechanism, environment-limitation note re: this session's sandbox having no real cgroup-backed worker to live-verify against) -- this entry covers only the subset of that finding split out as lower-priority. Unlike bug-1735's memory/cpu_time (which have existing-but-mismatched fields), memory_swap/tasks_current have NO receiving field in xce_host_runusage_t at all -- the worker's measurement is a complete dead end on the controller side, not a naming mismatch. The peak/efficiency fields (xce_job_runusage_t's own top-level fields) are simply never computed by anything, anywhere -- confirmed via grep, no partial implementation exists to build on.
|
||||
|
||||
## Fix
|
||||
PARTIALLY FIXED (2026-08-30). Of the two gaps this entry covers:
|
||||
(1) memory_swap/tasks_current and (2) peak_cpu_time-as-running-max -- 2026-08-30, 5c2c6664 (see git history).
|
||||
|
||||
(1) memory_swap/tasks_current -- FIXED. Added mem_swap (double)/tasks_current (int) fields to xce_host_runusage_t (types.h), matching capnp fields (memSwap @3/tasksCurrent @4) appended to the HostRunusage schema (controller.capnp) for WAL field-completeness, parsing in update_host_runusage() (misc.c) for the worker's 'memory_swap'/'tasks_current' resnames, and per-host copy in merge_job_runusage()'s (job.c) existing-host branch (the new fields would otherwise have been silently dropped on WAL replay/catchup for a host already seen).
|
||||
|
||||
(2) peak_cpu_time -- FIXED. aggregate_job_runusage() (misc.c) now tracks it as the running MAX of the per-call cpu_time sum across every report a job has ever received (not just the latest), guarding against a later report reducing it (e.g. an out-of-order merge/replay).
|
||||
|
||||
STILL NOT FIXED, DELIBERATELY (documented scope decision, not an oversight -- see aggregate_job_runusage()'s own doc comment in misc.c): peak_cpu_duration and the three efficiency fields (mem_efficiency/avg_cpu_efficiency/peak_cpu_efficiency). These need a 'used vs. requested' ratio against a job's resource request (a resreq expression tree, not a plain core/slot count) and/or an unspecified duration semantic -- genuine product/design decisions, not a formula that can be safely guessed. Left at their existing zero value rather than populated with an invented number that could mislead an operator.
|
||||
|
||||
Verified: full `meson test --suite controller` clean, Ok:6 Fail:0, 1220/1220 individual controller_unit_test cases pass (zero regression from touching the shared xce_host_runusage_t struct + its capnp schema), plus 2 new regression tests in test_host_runusage.c (7/7 cases pass).
|
||||
(3) 6df644e6, 2026-10-01: the remaining fields, with LSF bjobs -l definitions (chosen by the user). `update_job_cpu_stats()` (`src/controller/misc.c`), called by `update_job_rusages()` (`job.c`) after each report:
|
||||
- each host's `cpu_rate` = CPU-seconds gained / wall seconds since that host's previous report (`sample_time`); the job's cores = sum over its hosts. A report no later than the previous one, or with a CPU counter that went backwards, leaves the rate alone.
|
||||
- `peak_cpu` = highest job cores seen; `peak_cpu_time` = it rounded (the wire field is Int32); `peak_cpu_duration` = the interval that set it. This replaces the earlier `peak_cpu_time` = running max of cumulative CPU seconds, which only duplicated `cpu_time`.
|
||||
- `avg_cpu_efficiency` = cpu_time / elapsed since start / allocated slots; `peak_cpu_efficiency` = peak_cpu / slots; `mem_efficiency` = highest used_mem / the job-wide `rusage[mem]` request (`calc_wanted_job_total_resval`, per task by default, as the scheduler reserves it); all in percent, as the apiserver/xcecli models document; 0 when there is no slot count or no memory request (or the request is not a single primary one).
|
||||
New persisted fields `HostRunusage.sampleTime`/`cpuRate` and `JobRunusage.peakCpu` are appended to `controller.capnp` (old WAL entries still decode) and copied by `merge_job_runusage()`. The values are controller-side only: the apiserver does not map run usage into its job model and xcecli/harness do not display these fields.
|
||||
|
||||
## Verification
|
||||
_Not recorded (migrated from buglog.json)._
|
||||
- Test: `test_cpu_stats_*` (rate/peak/duration/efficiency, multi-host sum, mem efficiency as a peak, guards) and `test_aggregate_job_runusage_leaves_peak_cpu_alone` in `src/controller/testsuite/test_host_runusage.c` (11/11; written before `update_job_cpu_stats()` existed); WAL round trip of the new fields added to `test_wal_job_rusage_fact.c`, and the existing-host merge test `test_wru_apply_committed_entry_merges_without_freeing_live_host`: RED with the new `merge_job_runusage()` lines removed (`Expected 3.25 Was 0`), GREEN with them
|
||||
- Suites: controller_unit_test 1275/1275; `xce-build.sh all asan build.jai` 126/127 -- the one failure was the known intermittent harness pty timeout bug-1902 (recurrence recorded; 3/3 reruns pass)
|
||||
- Live: n/a (no consumer displays these fields; the computation is unit-tested against the same report sequence update_job_rusages() applies)
|
||||
|
||||
## References
|
||||
- Related: bug-1735
|
||||
|
|
|
|||
|
|
@ -1,13 +1,13 @@
|
|||
---
|
||||
id: "bug-1902"
|
||||
status: "cannot-reproduce"
|
||||
status_updated: "2026-10-01"
|
||||
status_note: "seen once during task 4.3; passed 5 subsequent runs"
|
||||
status: "investigating"
|
||||
status_updated: "2026-10-02"
|
||||
status_note: "recurred 2026-10-01 in a full xce-build all run (bug-1736 work): test_walcompare_unrecognized_flag_rejected_with_usage, 'xce_test did not exit within the harness's safety timeout after quit'; passed 3/3 targeted reruns"
|
||||
component: "harness"
|
||||
severity: "noise"
|
||||
timestamp: "2026-10-01T11:22:43Z"
|
||||
last_seen: "2026-10-01T11:22:43Z"
|
||||
occurrences: 1
|
||||
last_seen: "2026-10-02T04:14:22Z"
|
||||
occurrences: 2
|
||||
files: ["src/harness/testsuite/test_test_cli.c"]
|
||||
tags: ["harness", "test", "flaky", "pty"]
|
||||
related: []
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue