Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
12 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | |||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1690 | fixed | 2026-09-24 | controller | unknown | 2026-08-25T11:00:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1690: xce-audit-invariants finding: WAL_JOB_RUSAGE silently drops job.run_usage (client-reported…
Description
xce-audit-invariants finding: WAL_JOB_RUSAGE silently drops job.run_usage (client-reported per-host CPU/mem usage) on a controller restart replay -- same bug class as bug-1678 (WAL_JOB_DISPATCH/job.allocs), bug-1687 (WAL_JOB_START/job.pidinfo+start_time), bug-1689 (WAL_JOB_FINISH/job.exit_info+finish_time). update_job_rusages() (job.c) sets job->job.run_usage (xce_job_runusage_t*, an accumulated per-host struct) via update_host_runusage() (misc.c), but the WAL_JOB_RUSAGE fact persist_job_hook_fun (persist.c) wrote only carried wal_jobstate_t{jobid, state} -- run_usage was never in the log at all, so a restart replay left it permanently NULL even though a client actively reported CPU/mem usage. Unlike the three precedents, xce_job_runusage_t had NO existing Cap'n Proto mapping at all (confirmed: grep for xce_job_runusage_t in controller.capnp matched nothing; a similar-looking but unrelated type, xce_job_rusages_t/JobRusages/SchedJobRusages -- the CLIENT's wire update-request shape {jobid, pids[], pgids[], usages[]} -- already existed and is NOT interchangeable), so this fix required a genuine schema addition, not a reuse.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
persist_job_hook_fun()'s HOOK_DETAIL_JOB_RUSAGE_UPDATE case (persist.c) built only wal_jobstate_t{jobid, state} for the WAL_JOB_RUSAGE fact, discarding the job's run_usage entirely even though it was available (looked up via the same jobmgr lookup, under the same write lock update_job_rusages() already held). Both apply seams -- apply_committed_entry()'s shared WAL_JOB_START/FINISH/.../DISPATCH fall-through group (persist.c) and apply_wal_entry()'s identical shared group -> restore_job_state() (job.c, state-only) -- never restored run_usage, matching the exact same class of gap bug-1678/1687/1689 already fixed for allocs/pidinfo+start_time/exit_info+finish_time. Confirmed via CodeGraph source read of update_job_rusages() (job.c) and persist_job_hook_fun()'s actual HOOK_DETAIL_JOB_RUSAGE_UPDATE arm (not a direct persist_submit_job__fact() call site like DISPATCH/START/FINISH have -- RUSAGE is purely hook-driven from the raw xce_job_rusages_t payload update_job_rusages() fires, a genuine divergence from the audit's initial reconstructed precedent that was verified against the real code before implementing, per this task's explicit instruction).
Fix
FIXED (2026-08-25). Added a genuine new Cap'n Proto schema pair to controller.capnp -- struct HostRunusage $C.mapname("xce_host_runusage_t"){hostName, cpuTime, memUsage} and struct JobRunusage $C.mapname("xce_job_runusage_t"){maxMem, avgMem, usedMem, cpuTime, peakCpuTime, peakCpuDuration, memEfficiency, avgCpuEfficiency, peakCpuEfficiency, hostRunusage: List(HostRunusage) $C.maplistcount("num_hosts")} -- following JobPendReason's exact List(NestedStruct)+maplistcount precedent already in the schema, field types verified against types.h's real xce_job_runusage_t/xce_host_runusage_t definitions via CodeGraph before writing. The meson custom_target (src/lib/meson.build) regenerated lib/controller.capnp.c/.h automatically on rebuild -- verified via nm on the built controller_unit_test binary showing the full genuinely-codegen'd symbol set (new_JobRunusage, read/write_JobRunusage, encode/decode_JobRunusage, free_JobRunusage_ptr, get/set per-field accessors, and the equivalent HostRunusage set), never hand-written. persist.c gains capnp_encode_jobrunusage()/decode_jobrunusage() (mirroring capnp_encode_joballoc()/decode_joballoc() exactly). persist_job_hook_fun()'s HOOK_DETAIL_JOB_RUSAGE_UPDATE case now also encodes j->job.run_usage (when non-NULL) as trailing capnp bytes after the wal_jobstate_t header -- inlined into that hook (not a new persist_submit_job_rusage_fact(), unlike bug-1678/1687/1689's direct-call-site pattern, since RUSAGE is genuinely hook-driven). Both apply seams updated: apply_wal_entry() splits WAL_JOB_RUSAGE out of its shared state-only group into its own case, decoding trailing bytes and calling new restore_job_rusage_state() (job.c). apply_committed_entry()'s WAL_JOB_RUSAGE handling stays in its shared job-fact group but -- critically, UNLIKE START/FINISH/DISPATCH -- decodes/adopts run_usage UNCONDITIONALLY, before the JOB_TRANSITION_APPLIED gate: a rusage report almost never changes job.state (the job stays RUN across every repeat report), so gating adoption on a forward state transition the way the other three fixes do would make it dead code for the realistic, common case (every report after the first hits JOB_TRANSITION_NOOP_DUPLICATE and the gated block never runs). This divergence was caught by actually reading apply_job_update_transition()'s guard logic rather than blindly copying the precedent pattern. A payload of exactly sizeof(wal_jobstate_t) (any WAL_JOB_RUSAGE entry written before this fix, or one fired while job.run_usage was still NULL at write time) decodes no run_usage on both seams, matching pre-fix behavior exactly -- full backward compatibility.
CRITICAL SAFETY FINDING beyond the precedent's pattern, caught via investigation (not assumed): a naive free-and-replace (mirroring bug-1678/1687/1689's free_X_ptr()-then-adopt exactly) would be a genuine heap-corruption hazard here, specific to RUSAGE. job.run_usage can be the job's LIVE structure -- update_job_rusages()/update_host_runusage() (job.c/misc.c) mutate it synchronously and in place BEFORE the WAL fact is even appended, and host_name is sourced from makeString() (src/lib/xstring.c), an RTTI-managed, refcounted, header-before-data pointer (confirmed via librtti's actual source in the nix store: RTTI_DATA_TO_OBJ subtracts RTTI_OBJ_HEADER_SIZE from the data pointer to reach the real allocation start) -- NOT a plain malloc() allocation, unlike JobAlloc/JobPIDInfo's host/exdata fields (verified to be genuinely capnp-wire-decoded end to end via dispatch_a_job_()'s own alloc parameter, confirming why bug-1678/1687 never hit this). persist_apply_pump_run() (the live online-apply pump, confirmed via its own doc comment to run on the primary itself, not just boot replay) re-applies every committed WAL_JOB_RUSAGE fact via apply_committed_entry() moments after the SAME live mutation already happened -- so calling the capnp-generated free_JobRunusage_ptr() (written for plain malloc'd Text fields) on that live structure would free() a non-malloc'd, RTTI-managed pointer: guaranteed ASAN abort / heap corruption on the very first live rusage report for a new host. Fixed by making adoption a MERGE (new merge_job_runusage(), job.c) instead of a free-and-replace: an already-present host (matched by host_name string comparison, mirroring update_host_runusage()'s own find-existing-or-append semantics) only has its scalar cpu_time/mem_usage overwritten in place -- its host_name pointer, possibly xstring-backed, is NEVER touched or freed; only a genuinely new host's own freshly-decoded (always plain-malloc'd) host_name is adopted by ownership transfer. Applied identically in both restore_job_rusage_state() (apply_wal_entry seam) and apply_committed_entry's own branch, since a demoted-primary-turned-standby could in principle retain live-mutated run_usage data too. Noted (not fixed, genuinely out of scope -- a different file/component/bug class): job_destroy() (src/lib/jobj.c)'s own existing manual FREEUP-based walk over run_usage has this exact same latent host_name/xstring mismatch, independent of and pre-dating this fix.
Regression tests: src/controller/testsuite/test_wal_job_rusage_fact.c (8 tests, mirroring test_wal_job_start_fact.c's structure): (1) apply_committed_entry adopts run_usage when state does NOT change (JOB_TRANSITION_NOOP_DUPLICATE) -- the suite's most important test, proving the unconditional-decode design choice; (2) apply_committed_entry adopts run_usage on a genuine forward transition, hook still fires once; (3) apply_committed_entry backward-compat with an old-format payload; (4) apply_committed_entry merges without freeing a REAL makeString()'d host_name -- builds a live-flavor run_usage by hand, applies an update for that same host plus a new host, and asserts the host_name POINTER IDENTITY survives (proving it was never freed) alongside correct scalar/new-host merging, with careful test-only cleanup (detach + dereference()) so the separate, pre-existing job_destroy() FREEUP issue can't fire inside the test's own teardown; (5) apply_wal_entry adopts run_usage, no hook, absolute state overwrite; (6) apply_wal_entry backward-compat; (7) direct-payload restart-replay (WAL_JOB_SUBMIT + WAL_JOB_RUSAGE via persist_wal_async + persist_recover()); (8) drives the REAL, unmodified persist_job_hook_fun() via the established 'variadic trampoline' pattern (test_commit_barrier_rollout.c/test_persist_hooks.c) through a real WAL-write + persist_recover() restart cycle -- NOT via update_job_rusages() itself, because this same controller_unit_test binary already links test_commit_barrier_reply.c's own non-static stub of update_host_runusage() (a no-op, since misc.c is outside controller_unit_test_src), so calling update_job_rusages() here would silently hit that stub and populate nothing -- documented in the test file's own header comment.
RED->GREEN independently verified live (not narrated): git stash push -- src/controller/job.c src/controller/persist.c (keeping the schema change, the new test file, and meson.build/controller_test.c wiring), full clean rebuild via nix develop ./src --command bash src/tools/xce-build.sh controller asan build-asan (no --incremental, since this is the schema's first-ever compile) -- 1160 Tests 8 Failures: exactly the 6 new RUSAGE tests (5 with the exact expected 'must set job.run_usage' mismatch, 1 with 'Expected 500 Was 100' for the merge-safety test) plus the 2 pre-existing bug-1561 test_config_repl.c failures; the 2 backward-compat tests correctly passed unchanged. git stash pop, incremental rebuild -- 1160 Tests 2 Failures (only the pre-existing bug-1561 pair, exact signature match 'Expected 2 Was 3' / 'Expected 0 Was '), all 8 new tests pass, zero new failures, zero ASAN aborts/heap-corruption findings (explicitly grepped the full test log for AddressSanitizer/heap-buffer-overflow/invalid-free/SEGV/Aborted -- none found). Verified the generated capnp code was genuinely regenerated, not hand-written: nm on the built controller_unit_test binary shows the complete real codegen symbol set for both new structs (new_JobRunusage, read/write_JobRunusage, encode/decode_JobRunusage, free_JobRunusage_ptr, per-field get/set accessors, and the equivalent HostRunusage set), with generated-file mtimes strictly after the schema edit's mtime.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1678, bug-1687, bug-1689, bug-1561