fix(controller): WAL_JOB_RUSAGE loses job.run_usage on replay (bug-1690)

Same class as bug-1678/1687/1689: the WAL fact for WAL_JOB_RUSAGE only
carried {jobid, state}, so job.run_usage (client-reported per-host
CPU/mem usage) was never restored on either apply seam after a restart.

Unlike the prior three, xce_job_runusage_t/xce_host_runusage_t had no
existing Cap'n Proto mapping -- a genuine new schema addition
(HostRunusage/JobRunusage in controller.capnp, mirroring JobPendReason's
List(NestedStruct)+$C.maplistcount style), verified against the real
struct definitions in types.h, codegen confirmed via nm on the built
binary rather than assumed.

Two deliberate deviations from the established precedent, found by
reading the real code rather than copying the pattern blindly:

1. Adoption is unconditional, not gated on a state transition -- a
   rusage report almost never changes job.state (repeat reports keep
   state at RUN), so gating on JOB_TRANSITION_APPLIED like START/FINISH/
   DISPATCH do would make the fix dead code for the realistic case.

2. Merge, not free-and-replace. update_host_runusage() builds run_usage
   with host_name sourced from makeString() -- an RTTI-managed pointer,
   not malloc() -- and the primary's own live apply pump re-applies every
   fact it just wrote. A naive free_JobRunusage_ptr()-then-adopt (matching
   the prior three fixes exactly) would free() a non-malloc'd pointer on
   the very first live rusage report for a new host -- guaranteed heap
   corruption. Fixed with merge_job_runusage(): existing hosts get
   scalar-only in-place updates (host_name pointer never touched), only
   genuinely new hosts adopt their freshly-decoded (safe, malloc'd)
   host_name. Proved via a real makeString()-backed test asserting
   pointer identity survives, not just value equality. job_destroy()
   (src/lib/jobj.c) was found to have this identical latent mismatch,
   pre-existing and independent of this fix -- flagged, not fixed here.

Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension). 8 new tests in test_wal_job_rusage_fact.c; RED/GREEN
independently verified. Full controller suite: 1160 tests, 2 failures --
both the pre-existing, unrelated bug-1561 overlayfs sandbox flake. No
ASAN/heap-corruption findings. Backward compatibility confirmed for
old-format bare-payload entries.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg

src/controller/job.c

@@ -890,6 +890,139 @@ int restore_job_finish_state(cluster_t* c, uint64_t jobid, uint32_t state,
     return 0;
 }
 
+/* bug-1690: merge (never wholesale free-and-replace) a freshly-decoded
+ * xce_job_runusage_t into *dst. Takes ownership of `src` -- always
+ * consumed/freed by this call (either adopted whole, partially cannibalized
+ * for its new-host entries, or fully released), regardless of outcome.
+ *
+ * WHY A MERGE, NOT A REPLACE: *dst can be the job's LIVE run_usage --
+ * update_job_rusages() (job.c) mutates it synchronously and in place, via
+ * update_host_runusage() (misc.c), BEFORE the corresponding WAL fact is even
+ * appended. On the primary, persist_apply_pump_run() (persist.c) later
+ * re-applies that SAME fact via apply_committed_entry() moments afterward,
+ * on the SAME live job object -- so *dst can genuinely be that live,
+ * already-current structure at the moment this runs, not just a stale
+ * replay artifact. Critically, update_host_runusage() sources each
+ * xce_host_runusage_t.host_name from makeString() (src/lib/xstring.c) -- an
+ * RTTI-managed, refcounted intern-table pointer, NOT a plain malloc()
+ * allocation. RTTI's own header-before-data layout (librtti's
+ * RTTI_DATA_TO_OBJ: the object header sits BEFORE the pointer callers hold)
+ * means a plain free() on that pointer -- which the capnp-generated
+ * free_JobRunusage_ptr()/free_HostRunusage_ptr() would do, being written for
+ * genuinely capnp-decoded, plain-malloc'd Text fields -- corrupts the heap.
+ * (job_destroy()'s own manual FREEUP-based walk over run_usage, jobj.c, has
+ * this exact same latent mismatch already, independent of this fix -- flagged
+ * separately, not fixed here, since jobj.c/misc.c are outside this fix's
+ * scope.)
+ *
+ * The merge below NEVER frees an existing host_name: an already-present
+ * host (matched by name) only has its scalar cpu_time/mem_usage overwritten
+ * in place; only a genuinely NEW host's own freshly-decoded (always
+ * plain-malloc'd) host_name is adopted, by ownership transfer, never a
+ * live/xstring one. `src`'s own shell plus every entry NOT transferred is
+ * released via the ordinary capnp free path at the end -- safe, since `src`
+ * is always capnp-decoded, end to end, regardless of what *dst was. */
+void merge_job_runusage(xce_job_runusage_t** dst, xce_job_runusage_t* src)
+{
+    xce_job_runusage_t* d;
+    int i, k;
+
+    if (*dst == NULL) {
+        /* Nothing live to preserve — adopt the decoded copy outright,
+         * exactly like every other restore_job_*_state() sibling. */
+        *dst = src;
+        return;
+    }
+
+    d = *dst;
+    /* Scalar job-level fields: no pointers, always safe to overwrite in
+     * place regardless of *dst's provenance. */
+    d->max_mem            = src->max_mem;
+    d->avg_mem            = src->avg_mem;
+    d->used_mem           = src->used_mem;
+    d->cpu_time            = src->cpu_time;
+    d->peak_cpu_time       = src->peak_cpu_time;
+    d->peak_cpu_duration   = src->peak_cpu_duration;
+    d->mem_efficiency      = src->mem_efficiency;
+    d->avg_cpu_efficiency  = src->avg_cpu_efficiency;
+    d->peak_cpu_efficiency = src->peak_cpu_efficiency;
+
+    for (i = 0; i < src->num_hosts; i++) {
+        xce_host_runusage_t* sh      = src->host_runusage[i];
+        int                  matched = -1;
+
+        if (sh == NULL)
+            continue;
+
+        for (k = 0; k < d->num_hosts; k++) {
+            if (d->host_runusage[k] != NULL &&
+                d->host_runusage[k]->host_name != NULL &&
+                sh->host_name != NULL &&
+                strcmp(sh->host_name, d->host_runusage[k]->host_name) == 0) {
+                matched = k;
+                break;
+            }
+        }
+
+        if (matched >= 0) {
+            /* Existing host: overwrite scalars only. d's host_name pointer
+             * (possibly xstring-backed) is never touched or freed here. */
+            d->host_runusage[matched]->cpu_time  = sh->cpu_time;
+            d->host_runusage[matched]->mem_usage = sh->mem_usage;
+        } else {
+            /* Genuinely new host: append, transferring ownership of sh's
+             * own plain-malloc'd host_name directly -- never xstring. */
+            xce_host_runusage_t** grown = (xce_host_runusage_t**)realloc(
+                d->host_runusage,
+                (size_t)(d->num_hosts + 1) * sizeof(xce_host_runusage_t*));
+            if (grown == NULL)
+                continue; /* OOM -- skip this host, keep the rest merged */
+            d->host_runusage             = grown;
+            d->host_runusage[d->num_hosts] = sh;
+            d->num_hosts++;
+            src->host_runusage[i] = NULL; /* ownership transferred -- the
+                                           * free_JobRunusage_ptr() call
+                                           * below must not also free it */
+        }
+    }
+
+    /* Release src's shell + every entry whose ownership was NOT transferred
+     * above -- always safe, src is capnp-decoded end to end. */
+    free_JobRunusage_ptr(&src);
+}
+
+/* bug-1690: WAL_JOB_RUSAGE's own sibling of restore_job_state() above, for
+ * apply_wal_entry()'s standby catch-up / replicated-apply seam -- takes
+ * ownership of an optional decoded xce_job_runusage_t (NULL for a
+ * state-only WAL_JOB_RUSAGE entry -- either one written before this fix, or
+ * one fired while job.run_usage was still NULL at write time) so a promoted
+ * standby recovers the client-reported per-host CPU/mem usage the same way
+ * apply_committed_entry's WAL_JOB_RUSAGE case already does for the primary's
+ * own boot replay. Unconditional absolute state overwrite, matching every
+ * other restore_job_*_state() sibling on this seam -- no transition-result
+ * guarding happens on standby catch-up. Merges (never frees-and-replaces)
+ * run_usage -- see merge_job_runusage()'s doc comment above for why. */
+int restore_job_rusage_state(cluster_t* c, uint64_t jobid, uint32_t state,
+                             xce_job_runusage_t* runusage)
+{
+    job_t* j = NULL;
+    lock(c->jobmgr, RTTI_LOCK_READ);
+    key_find(c->jobmgr, (void*)jobid, (void**)&j);
+    unlock(c->jobmgr);
+    if (!j) {
+        if (runusage != NULL)
+            free_JobRunusage_ptr(&runusage);
+        return -ENOENT;
+    }
+
+    lock(j, RTTI_LOCK_WRITE);
+    j->job.state = (int)state;
+    if (runusage != NULL)
+        merge_job_runusage(&(j->job.run_usage), runusage);
+    unlock(j);
+    return 0;
+}
+
 static int get_nslots_in_alloc(xce_host_alloc_t* alloc)
 {
     int i;