---
id: "bug-1690"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-08-25T11:00:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/schema/controller.capnp (new HostRunusage/JobRunusage structs)", "src/controller/persist.c (capnp_encode_jobrunusage/decode_jobrunusage ~L327-388", "persist_job_hook_fun's HOOK_DETAIL_JOB_RUSAGE_UPDATE case ~L7440-7500", "apply_committed_entry's WAL_JOB_RUSAGE branch ~L4116-4144", "apply_wal_entry's WAL_JOB_RUSAGE case ~L3712-3740)", "src/controller/job.c (merge_job_runusage new ~L893-1000", "restore_job_rusage_state new)", "src/include/wal_types.h (WAL_JOB_RUSAGE doc comment)", "src/controller/testsuite/test_wal_job_rusage_fact.c (new", "8 tests)"]
tags: ["controller", "job.c", "persist.c", "wal", "wal-job-rusage", "job-run-usage", "restart-recovery", "data-loss", "apply-seam", "capnp-schema-change", "heap-corruption-hazard", "xstring-rtti-pointer", "merge-not-replace"]
related: ["bug-1678", "bug-1687", "bug-1689", "bug-1561"]
legacy: true
---

# bug-1690: xce-audit-invariants finding: WAL_JOB_RUSAGE silently drops job.run_usage (client-reported…

## Description
xce-audit-invariants finding: WAL_JOB_RUSAGE silently drops job.run_usage (client-reported per-host CPU/mem usage) on a controller restart replay -- same bug class as bug-1678 (WAL_JOB_DISPATCH/job.allocs), bug-1687 (WAL_JOB_START/job.pidinfo+start_time), bug-1689 (WAL_JOB_FINISH/job.exit_info+finish_time). update_job_rusages() (job.c) sets job->job.run_usage (xce_job_runusage_t*, an accumulated per-host struct) via update_host_runusage() (misc.c), but the WAL_JOB_RUSAGE fact persist_job_hook_fun (persist.c) wrote only carried wal_jobstate_t{jobid, state} -- run_usage was never in the log at all, so a restart replay left it permanently NULL even though a client actively reported CPU/mem usage. Unlike the three precedents, xce_job_runusage_t had NO existing Cap'n Proto mapping at all (confirmed: grep for xce_job_runusage_t in controller.capnp matched nothing; a similar-looking but unrelated type, xce_job_rusages_t/JobRusages/SchedJobRusages -- the CLIENT's wire update-request shape {jobid, pids[], pgids[], usages[]} -- already existed and is NOT interchangeable), so this fix required a genuine schema addition, not a reuse.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
persist_job_hook_fun()'s HOOK_DETAIL_JOB_RUSAGE_UPDATE case (persist.c) built only wal_jobstate_t{jobid, state} for the WAL_JOB_RUSAGE fact, discarding the job's run_usage entirely even though it was available (looked up via the same jobmgr lookup, under the same write lock update_job_rusages() already held). Both apply seams -- apply_committed_entry()'s shared WAL_JOB_START/FINISH/.../DISPATCH fall-through group (persist.c) and apply_wal_entry()'s identical shared group -> restore_job_state() (job.c, state-only) -- never restored run_usage, matching the exact same class of gap bug-1678/1687/1689 already fixed for allocs/pidinfo+start_time/exit_info+finish_time. Confirmed via CodeGraph source read of update_job_rusages() (job.c) and persist_job_hook_fun()'s actual HOOK_DETAIL_JOB_RUSAGE_UPDATE arm (not a direct persist_submit_job_*_fact() call site like DISPATCH/START/FINISH have -- RUSAGE is purely hook-driven from the raw xce_job_rusages_t* payload update_job_rusages() fires, a genuine divergence from the audit's initial reconstructed precedent that was verified against the real code before implementing, per this task's explicit instruction).

## Fix
FIXED (2026-08-25). Added a genuine new Cap'n Proto schema pair to controller.capnp -- struct HostRunusage $C.mapname("xce_host_runusage_t"){hostName, cpuTime, memUsage} and struct JobRunusage $C.mapname("xce_job_runusage_t"){maxMem, avgMem, usedMem, cpuTime, peakCpuTime, peakCpuDuration, memEfficiency, avgCpuEfficiency, peakCpuEfficiency, hostRunusage: List(HostRunusage) $C.maplistcount("num_hosts")} -- following JobPendReason's exact List(NestedStruct)+maplistcount precedent already in the schema, field types verified against types.h's real xce_job_runusage_t/xce_host_runusage_t definitions via CodeGraph before writing. The meson custom_target (src/lib/meson.build) regenerated lib/controller.capnp.c/.h automatically on rebuild -- verified via nm on the built controller_unit_test binary showing the full genuinely-codegen'd symbol set (new_JobRunusage, read/write_JobRunusage, encode/decode_JobRunusage, free_JobRunusage_ptr, get/set per-field accessors, and the equivalent HostRunusage set), never hand-written. persist.c gains capnp_encode_jobrunusage()/decode_jobrunusage() (mirroring capnp_encode_joballoc()/decode_joballoc() exactly). persist_job_hook_fun()'s HOOK_DETAIL_JOB_RUSAGE_UPDATE case now also encodes j->job.run_usage (when non-NULL) as trailing capnp bytes after the wal_jobstate_t header -- inlined into that hook (not a new persist_submit_job_rusage_fact(), unlike bug-1678/1687/1689's direct-call-site pattern, since RUSAGE is genuinely hook-driven). Both apply seams updated: apply_wal_entry() splits WAL_JOB_RUSAGE out of its shared state-only group into its own case, decoding trailing bytes and calling new restore_job_rusage_state() (job.c). apply_committed_entry()'s WAL_JOB_RUSAGE handling stays in its shared job-fact group but -- critically, UNLIKE START/FINISH/DISPATCH -- decodes/adopts run_usage UNCONDITIONALLY, before the JOB_TRANSITION_APPLIED gate: a rusage report almost never changes job.state (the job stays RUN across every repeat report), so gating adoption on a forward state transition the way the other three fixes do would make it dead code for the realistic, common case (every report after the first hits JOB_TRANSITION_NOOP_DUPLICATE and the gated block never runs). This divergence was caught by actually reading apply_job_update_transition()'s guard logic rather than blindly copying the precedent pattern. A payload of exactly sizeof(wal_jobstate_t) (any WAL_JOB_RUSAGE entry written before this fix, or one fired while job.run_usage was still NULL at write time) decodes no run_usage on both seams, matching pre-fix behavior exactly -- full backward compatibility.

CRITICAL SAFETY FINDING beyond the precedent's pattern, caught via investigation (not assumed): a naive free-and-replace (mirroring bug-1678/1687/1689's free_X_ptr()-then-adopt exactly) would be a genuine heap-corruption hazard here, specific to RUSAGE. job.run_usage can be the job's LIVE structure -- update_job_rusages()/update_host_runusage() (job.c/misc.c) mutate it synchronously and in place BEFORE the WAL fact is even appended, and host_name is sourced from makeString() (src/lib/xstring.c), an RTTI-managed, refcounted, header-before-data pointer (confirmed via librtti's actual source in the nix store: RTTI_DATA_TO_OBJ subtracts RTTI_OBJ_HEADER_SIZE from the data pointer to reach the real allocation start) -- NOT a plain malloc() allocation, unlike JobAlloc/JobPIDInfo's host/exdata fields (verified to be genuinely capnp-wire-decoded end to end via dispatch_a_job_()'s own alloc parameter, confirming why bug-1678/1687 never hit this). persist_apply_pump_run() (the live online-apply pump, confirmed via its own doc comment to run on the primary itself, not just boot replay) re-applies every committed WAL_JOB_RUSAGE fact via apply_committed_entry() moments after the SAME live mutation already happened -- so calling the capnp-generated free_JobRunusage_ptr() (written for plain malloc'd Text fields) on that live structure would free() a non-malloc'd, RTTI-managed pointer: guaranteed ASAN abort / heap corruption on the very first live rusage report for a new host. Fixed by making adoption a MERGE (new merge_job_runusage(), job.c) instead of a free-and-replace: an already-present host (matched by host_name string comparison, mirroring update_host_runusage()'s own find-existing-or-append semantics) only has its scalar cpu_time/mem_usage overwritten in place -- its host_name pointer, possibly xstring-backed, is NEVER touched or freed; only a genuinely new host's own freshly-decoded (always plain-malloc'd) host_name is adopted by ownership transfer. Applied identically in both restore_job_rusage_state() (apply_wal_entry seam) and apply_committed_entry's own branch, since a demoted-primary-turned-standby could in principle retain live-mutated run_usage data too. Noted (not fixed, genuinely out of scope -- a different file/component/bug class): job_destroy() (src/lib/jobj.c)'s own existing manual FREEUP-based walk over run_usage has this exact same latent host_name/xstring mismatch, independent of and pre-dating this fix.

Regression tests: src/controller/testsuite/test_wal_job_rusage_fact.c (8 tests, mirroring test_wal_job_start_fact.c's structure): (1) apply_committed_entry adopts run_usage when state does NOT change (JOB_TRANSITION_NOOP_DUPLICATE) -- the suite's most important test, proving the unconditional-decode design choice; (2) apply_committed_entry adopts run_usage on a genuine forward transition, hook still fires once; (3) apply_committed_entry backward-compat with an old-format payload; (4) apply_committed_entry merges without freeing a REAL makeString()'d host_name -- builds a live-flavor run_usage by hand, applies an update for that same host plus a new host, and asserts the host_name POINTER IDENTITY survives (proving it was never freed) alongside correct scalar/new-host merging, with careful test-only cleanup (detach + dereference()) so the separate, pre-existing job_destroy() FREEUP issue can't fire inside the test's own teardown; (5) apply_wal_entry adopts run_usage, no hook, absolute state overwrite; (6) apply_wal_entry backward-compat; (7) direct-payload restart-replay (WAL_JOB_SUBMIT + WAL_JOB_RUSAGE via persist_wal_async + persist_recover()); (8) drives the REAL, unmodified persist_job_hook_fun() via the established 'variadic trampoline' pattern (test_commit_barrier_rollout.c/test_persist_hooks.c) through a real WAL-write + persist_recover() restart cycle -- NOT via update_job_rusages() itself, because this same controller_unit_test binary already links test_commit_barrier_reply.c's own non-static stub of update_host_runusage() (a no-op, since misc.c is outside controller_unit_test_src), so calling update_job_rusages() here would silently hit that stub and populate nothing -- documented in the test file's own header comment.

RED->GREEN independently verified live (not narrated): `git stash push -- src/controller/job.c src/controller/persist.c` (keeping the schema change, the new test file, and meson.build/controller_test.c wiring), full clean rebuild via `nix develop ./src --command bash src/tools/xce-build.sh controller asan build-asan` (no --incremental, since this is the schema's first-ever compile) -- 1160 Tests 8 Failures: exactly the 6 new RUSAGE tests (5 with the exact expected 'must set job.run_usage' mismatch, 1 with 'Expected 500 Was 100' for the merge-safety test) plus the 2 pre-existing bug-1561 test_config_repl.c failures; the 2 backward-compat tests correctly passed unchanged. `git stash pop`, incremental rebuild -- 1160 Tests 2 Failures (only the pre-existing bug-1561 pair, exact signature match 'Expected 2 Was 3' / 'Expected 0 Was <garbage>'), all 8 new tests pass, zero new failures, zero ASAN aborts/heap-corruption findings (explicitly grepped the full test log for AddressSanitizer/heap-buffer-overflow/invalid-free/SEGV/Aborted -- none found). Verified the generated capnp code was genuinely regenerated, not hand-written: `nm` on the built controller_unit_test binary shows the complete real codegen symbol set for both new structs (new_JobRunusage, read/write_JobRunusage, encode/decode_JobRunusage, free_JobRunusage_ptr, per-field get/set accessors, and the equivalent HostRunusage set), with generated-file mtimes strictly after the schema edit's mtime.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1678, bug-1687, bug-1689, bug-1561