lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): WAL_JOB_RUSAGE loses job.run_usage on replay (bug-1690)

Browse source 
Same class as bug-1678/1687/1689: the WAL fact for WAL_JOB_RUSAGE only
carried {jobid, state}, so job.run_usage (client-reported per-host
CPU/mem usage) was never restored on either apply seam after a restart.

Unlike the prior three, xce_job_runusage_t/xce_host_runusage_t had no
existing Cap'n Proto mapping -- a genuine new schema addition
(HostRunusage/JobRunusage in controller.capnp, mirroring JobPendReason's
List(NestedStruct)+$C.maplistcount style), verified against the real
struct definitions in types.h, codegen confirmed via nm on the built
binary rather than assumed.

Two deliberate deviations from the established precedent, found by
reading the real code rather than copying the pattern blindly:

1. Adoption is unconditional, not gated on a state transition -- a
   rusage report almost never changes job.state (repeat reports keep
   state at RUN), so gating on JOB_TRANSITION_APPLIED like START/FINISH/
   DISPATCH do would make the fix dead code for the realistic case.

2. Merge, not free-and-replace. update_host_runusage() builds run_usage
   with host_name sourced from makeString() -- an RTTI-managed pointer,
   not malloc() -- and the primary's own live apply pump re-applies every
   fact it just wrote. A naive free_JobRunusage_ptr()-then-adopt (matching
   the prior three fixes exactly) would free() a non-malloc'd pointer on
   the very first live rusage report for a new host -- guaranteed heap
   corruption. Fixed with merge_job_runusage(): existing hosts get
   scalar-only in-place updates (host_name pointer never touched), only
   genuinely new hosts adopt their freshly-decoded (safe, malloc'd)
   host_name. Proved via a real makeString()-backed test asserting
   pointer identity survives, not just value equality. job_destroy()
   (src/lib/jobj.c) was found to have this identical latent mismatch,
   pre-existing and independent of this fix -- flagged, not fixed here.

Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension). 8 new tests in test_wal_job_rusage_fact.c; RED/GREEN
independently verified. Full controller suite: 1160 tests, 2 failures --
both the pre-existing, unrelated bug-1561 overlayfs sandbox flake. No
ASAN/heap-corruption findings. Backward compatibility confirmed for
old-format bare-payload entries.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 10:53:32 +00:00
parent c3d6cfa4cc
commit f7b732a300
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
8 changed files with 1328 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

29
src/schema/controller.capnp
View file

@ -266,6 +266,35 @@ struct JobPIDInfo $C.mapname("job_pidinfo_t"){
exdata @3: Text $C.mapname("exdata");
}
# bug-1690: xce_job_runusage_t (xce/types.h) previously had NO Cap'n Proto
# mapping at all -- unlike JobAlloc/JobPIDInfo above, which already existed
# as Job sub-fields the WAL_JOB_DISPATCH/WAL_JOB_START fixes (bug-1678/1687)
# could reuse standalone. WAL_JOB_RUSAGE needs one to carry the client-
# reported per-host CPU/mem usage in its fact payload so a restart/catch-up
# replay can reconstruct job.run_usage instead of leaving it permanently
# NULL. NOT the same type as JobRusages/SchedJobRusages below (xce_job_
# rusages_t) -- that is the client's WIRE UPDATE REQUEST shape
# ({jobid, pids[], pgids[], usages[]}); this is the CONTROLLER'S ACCUMULATED
# per-job/per-host usage shape, a different struct entirely.
struct HostRunusage $C.mapname("xce_host_runusage_t"){
hostName @0: Text $C.mapname("host_name");
cpuTime @1: Int32 $C.mapname("cpu_time");
memUsage @2: Float64 $C.mapname("mem_usage");
}
struct JobRunusage $C.mapname("xce_job_runusage_t"){
maxMem @0: Float64 $C.mapname("max_mem");
avgMem @1: Float64 $C.mapname("avg_mem");
usedMem @2: Float64 $C.mapname("used_mem");
cpuTime @3: Int32 $C.mapname("cpu_time");
peakCpuTime @4: Int32 $C.mapname("peak_cpu_time");
peakCpuDuration @5: Int32 $C.mapname("peak_cpu_duration");
memEfficiency @6: Float32 $C.mapname("mem_efficiency");
avgCpuEfficiency @7: Float32 $C.mapname("avg_cpu_efficiency");
peakCpuEfficiency @8: Float32 $C.mapname("peak_cpu_efficiency");
hostRunusage @9: List(HostRunusage) $C.mapname("host_runusage") $C.maplistcount("num_hosts");
}
struct JobFail $C.mapname("job_failinfo_t"){
jobId @0: UInt64 $C.mapname("jobid");
code @1: Int32 $C.mapname("code");