fix(controller): WAL_JOB_PREASON loses job.pend_reasons on replay (bug-1692)

Last of the five field-parity gaps from the original audit (bug-1678,
bug-1687, bug-1689, bug-1690, bug-1691). set_preason()/clear_preason()
set job.pend_reasons (host/resource-based pending-job explanations), but
persist_job_hook_fun's default: arm only ever captured {jobid, state}.
Neither apply seam restored pend_reasons after a restart.

Reuses the ALREADY-EXISTING JobPendReason/HostBasedReason/ResourceBasedReason
capnp structs -- no schema change needed, unlike bug-1690's WAL_JOB_RUSAGE.
Safety-checked against bug-1690's lesson (RTTI/makeString()-backed fields
need merge-not-replace, not free-and-replace): pend_reasons is plain
malloc/strdup-backed throughout (erase_job_preason is FREEUP()-only,
add_host_preason/add_res_preason use strdup()/calloc()), so a
straightforward free-and-replace is safe here, matching bug-1678/1687's
pattern instead.

New wrinkle beyond the four prior fixes: pend_reasons has a genuine CLEAR
operation, and "trailing bytes present or absent" can't distinguish a
legitimate clear from an old-format payload -- added an explicit presence
byte (0=cleared/adopt NULL, 1=capnp bytes follow), verified by two
dedicated "clear survives restart" tests.

11 new tests in test_wal_job_preason_fact.c; RED/GREEN independently
verified (7 genuine pre-fix failures, one test's own wrong premise about
capnp's garbage-byte behavior caught and corrected during RED/GREEN
iteration). Full controller suite: 1178 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake. No ASAN findings.

This closes the field-complete apply parity invariant class the
xce-audit-invariants controller dimension 7 was added for.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg

src/controller/job.c

@@ -1067,6 +1067,55 @@ int restore_job_ckpt_state(cluster_t* c, uint64_t jobid, uint32_t state,
     return 0;
 }
 
+/* bug-1692: WAL_JOB_PREASON's own sibling of restore_job_ckpt_state() above,
+ * for apply_wal_entry()'s standby catch-up / replicated-apply seam.
+ *
+ * has_field distinguishes a pre-fix bare {jobid,state} payload (has_field=0,
+ * pend_reasons left untouched -- full backward compatibility) from a
+ * post-fix payload that explicitly carries pend_reasons' new value
+ * (has_field=1): preason non-NULL adopts a freshly-decoded value, preason
+ * NULL means the field was genuinely cleared (do_clr_preason) and
+ * pend_reasons should be reset to NULL too -- pend_reasons is the one
+ * field-parity fact type in this family (bug-1678/1687/1689/1690/1691) with
+ * a real CLEAR operation, so unlike restore_job_ckpt_state()/
+ * restore_job_rusage_state() a NULL decoded value is NOT automatically
+ * "nothing to adopt".
+ *
+ * preason (when non-NULL) is always freshly heap-allocated by
+ * decode_jobpreason() (persist.c) -- plain calloc()/strdup() throughout
+ * (verified via erase_job_preason()'s FREEUP-only release and every
+ * add_*_preason() constructor in src/lib/preason.c, plus the
+ * capnp-generated decode_HostBasedReason/decode_ResourceBasedReason using
+ * STRING_DUP+calloc), NOT xstring/RTTI-backed like ckptinfo or rusage's
+ * host_name -- so a straightforward free-old-then-adopt-new via
+ * free_JobPendReason_ptr() is safe here, no merge-not-replace treatment
+ * needed (unlike bug-1690's WAL_JOB_RUSAGE). Ownership of preason always
+ * transfers to job->job.pend_reasons on success; released here if the job
+ * is gone. */
+int restore_job_preason_state(cluster_t* c, uint64_t jobid, uint32_t state,
+                              int has_field, xce_job_preason_t* preason)
+{
+    job_t* j = NULL;
+    lock(c->jobmgr, RTTI_LOCK_READ);
+    key_find(c->jobmgr, (void*)jobid, (void**)&j);
+    unlock(c->jobmgr);
+    if (!j) {
+        if (preason != NULL)
+            free_JobPendReason_ptr(&preason);
+        return -ENOENT;
+    }
+
+    lock(j, RTTI_LOCK_WRITE);
+    j->job.state = (int)state;
+    if (has_field) {
+        if (j->job.pend_reasons != NULL)
+            free_JobPendReason_ptr(&(j->job.pend_reasons));
+        j->job.pend_reasons = preason; /* NULL (cleared) or the decoded value */
+    }
+    unlock(j);
+    return 0;
+}
+
 static int get_nslots_in_alloc(xce_host_alloc_t* alloc)
 {
     int i;

src/controller/meson.build

@@ -174,6 +174,7 @@ controller_unit_test_src = [
   'testsuite/test_wal_job_finish_fact.c',
   'testsuite/test_wal_job_rusage_fact.c',
   'testsuite/test_wal_job_ckpt_fact.c',
+  'testsuite/test_wal_job_preason_fact.c',
   'testsuite/test_job_dispatch_fail_count.c',
   'testsuite/test_apply_reapply_idempotency.c',
   'testsuite/test_apply_correctness_verification.c',

src/controller/persist.c

@@ -84,6 +84,19 @@ extern int restore_job_rusage_state(cluster_t* c, uint64_t jobid,
  * restores it on the primary's own boot replay. */
 extern int restore_job_ckpt_state(cluster_t* c, uint64_t jobid,
                                   uint32_t state, char* ckptinfo);
+/* restore_job_preason_state defined in job.c (bug-1692): WAL_JOB_PREASON's own
+ * sibling of restore_job_state() above. Unlike restore_job_ckpt_state()/
+ * restore_job_rusage_state(), a NULL decoded pointer is not automatically
+ * "leave untouched" here -- pend_reasons has a genuine CLEAR operation
+ * (do_clr_preason/HOOK_DETAIL_CLRPREASON) that must durably survive replay
+ * too, so has_field disambiguates a pre-fix bare {jobid,state} payload
+ * (has_field=0, pend_reasons left untouched -- full backward compatibility)
+ * from a post-fix payload that explicitly carries the field's new value
+ * (has_field=1): preason non-NULL adopts a freshly-decoded value, preason
+ * NULL means pend_reasons was genuinely cleared and must become NULL too. */
+extern int restore_job_preason_state(cluster_t* c, uint64_t jobid,
+                                     uint32_t state, int has_field,
+                                     xce_job_preason_t* preason);
 /* merge_job_runusage defined in job.c (bug-1690): merges a freshly-decoded
  * xce_job_runusage_t into *dst rather than freeing-and-replacing it. Needed
  * here too (apply_committed_entry's own WAL_JOB_RUSAGE branch below), not
@@ -253,6 +266,67 @@ static xce_job_alloc_t* decode_joballoc(const uint8_t* payload, size_t len)
     return alloc;
 }
 
+/* Serialize xce_job_preason_t to a malloc'd Cap'n Proto buffer (bug-1692):
+ * lets WAL_JOB_PREASON carry the scheduler's host/resource-based pend
+ * reasons alongside the {jobid, state} header, so a restart replay can
+ * reconstruct job.pend_reasons instead of leaving it permanently NULL.
+ * Mirrors capnp_encode_joballoc above exactly, using the JobPendReason
+ * struct (already in controller.capnp as the existing Job.pendreason field
+ * -- no schema change needed here). */
+static int capnp_encode_jobpreason(const xce_job_preason_t* pr, uint8_t** out,
+                                   size_t* out_len)
+{
+    struct capn           c;
+    struct capn_segment*  cs;
+    struct JobPendReason  cjp;
+    JobPendReason_ptr     jpp;
+
+    capn_init_malloc(&c);
+    cs = capn_root(&c).seg;
+    encode_JobPendReason(cs, &cjp, (xce_job_preason_t*)pr);
+    jpp = new_JobPendReason(cs);
+    write_JobPendReason(&cjp, jpp);
+    capn_setp(capn_root(&c), 0, jpp.p);
+
+    *out_len = capn_size(&c);
+    *out     = malloc(*out_len);
+    if (!*out) { capn_free(&c); return -ENOMEM; }
+    capn_write_mem(&c, *out, *out_len, 0);
+    capn_free(&c);
+    return 0;
+}
+
+/* Decode a Cap'n Proto JobPendReason buffer (bug-1692) into a freshly heap-
+ * allocated xce_job_preason_t*, mirroring decode_joballoc()'s pattern above.
+ * Returns NULL on a NULL/empty payload; caller owns the result. The decoded
+ * tree is plain calloc()/strdup() throughout (capnp-generated
+ * decode_HostBasedReason/decode_ResourceBasedReason use STRING_DUP + calloc,
+ * verified in controller.capnp.c), NOT xstring/RTTI-backed like ckptinfo or
+ * rusage's host_name -- release via free_JobPendReason_ptr (plain free()),
+ * never dereference(). */
+static xce_job_preason_t* decode_jobpreason(const uint8_t* payload,
+                                            size_t len)
+{
+    struct capn          c_ctx;
+    JobPendReason_ptr    jpp;
+    struct JobPendReason cjp;
+    xce_job_preason_t*   preason;
+
+    if (!payload || len == 0)
+        return NULL;
+
+    capn_init_mem(&c_ctx, payload, len, 0);
+    jpp.p = capn_getp(capn_root(&c_ctx), 0, 1);
+    memset(&cjp, 0, sizeof(cjp));
+    read_JobPendReason(&cjp, jpp);
+
+    preason = calloc(1, sizeof(*preason));
+    if (!preason) { capn_free(&c_ctx); return NULL; }
+    decode_JobPendReason(preason, &cjp);
+    capn_free(&c_ctx);
+    return preason;
+}
+
 /* Serialize job_pidinfo_t to a malloc'd Cap'n Proto buffer (bug-1687):
  * lets WAL_JOB_START carry the worker-reported pids/pgids/exdata alongside
  * the {jobid, state} header, so a restart replay can reconstruct
@@ -3746,7 +3820,6 @@ int apply_wal_entry(persist_t* p, uint32_t type, uint64_t lsn,
     case WAL_JOB_STOP:
     case WAL_JOB_RESUME:
     case WAL_JOB_UPDATE:
-    case WAL_JOB_PREASON:
         /* Payload: wal_jobstate_t {jobid, state} */
         if (payload_len >= sizeof(wal_jobstate_t)) {
             const wal_jobstate_t* js = (const wal_jobstate_t*)payload;
@@ -3754,6 +3827,46 @@ int apply_wal_entry(persist_t* p, uint32_t type, uint64_t lsn,
         }
         break;
 
+    case WAL_JOB_PREASON: {
+        /* bug-1692: split out of the shared group above -- mirrors
+         * WAL_JOB_CKPT's/WAL_JOB_RUSAGE's splits (bug-1690/1691). Unlike
+         * those, pend_reasons has a genuine "clear" operation
+         * (do_clr_preason), so the payload beyond the wal_jobstate_t header
+         * always starts with an explicit presence byte (0 = cleared, adopt
+         * NULL; 1 = capnp JobPendReason bytes follow, adopt the decoded
+         * value) on any entry written after this fix. A payload of exactly
+         * sizeof(wal_jobstate_t) (no presence byte -- any WAL_JOB_PREASON
+         * entry written before this fix) leaves pend_reasons untouched,
+         * matching the pre-fix state-only behavior exactly. */
+        if (payload_len >= sizeof(wal_jobstate_t)) {
+            const wal_jobstate_t* js = (const wal_jobstate_t*)payload;
+            int                 has_field = 0;
+            xce_job_preason_t*  preason   = NULL;
+            if (payload_len > sizeof(wal_jobstate_t)) {
+                uint8_t present =
+                    *((const uint8_t*)payload + sizeof(wal_jobstate_t));
+                if (present) {
+                    if (payload_len > sizeof(wal_jobstate_t) + 1) {
+                        preason = decode_jobpreason(
+                            (const uint8_t*)payload + sizeof(wal_jobstate_t)
+                                + 1,
+                            payload_len - sizeof(wal_jobstate_t) - 1);
+                    }
+                    /* present-but-failed-decode (corrupt/truncated trailing
+                     * bytes) must NOT be treated as a clear -- leave
+                     * pend_reasons untouched, matching every other decode
+                     * helper's failure fallback in this function. */
+                    has_field = (preason != NULL);
+                } else {
+                    has_field = 1; /* explicit clear -- adopt NULL */
+                }
+            }
+            restore_job_preason_state(p->cluster, js->jobid, js->state,
+                                      has_field, preason);
+        }
+        break;
+    }
+
     case WAL_JOB_CKPT: {
         /* bug-1691: split out of the shared group above -- mirrors
          * WAL_JOB_RUSAGE's split just below (bug-1690). This is the one
@@ -4258,6 +4371,51 @@ int apply_committed_entry(persist_t* p, uint32_t wal_type, uint64_t lsn,
             }
         }
 
+        /* bug-1692: WAL_JOB_PREASON mirrors WAL_JOB_CKPT's/WAL_JOB_RUSAGE's
+         * unconditional-adopt pattern immediately above -- set_preason()/
+         * do_clr_preason() (job.c) never touch job.state, so gating
+         * adoption on result==JOB_TRANSITION_APPLIED would make it dead
+         * code: js->state is always the job's unchanged current state
+         * (persist_job_hook_fun's HOOK_DETAIL_SETPREASON/CLRPREASON arm
+         * captures it verbatim), so apply_job_update_transition() always
+         * reports JOB_TRANSITION_NOOP_DUPLICATE here. Decode/adopt
+         * unconditionally, before the transition-guarded block below.
+         *
+         * Unlike CKPT/RUSAGE, a presence byte (not "trailing bytes exist or
+         * not") disambiguates a genuine CLEAR (do_clr_preason) from an
+         * old-format payload -- see decode_jobpreason()'s/
+         * restore_job_preason_state()'s doc comments and WAL_JOB_PREASON's
+         * own doc comment in wal_types.h for the full rationale. `present ==
+         * 0` (explicit clear) or a successful decode both trigger adoption;
+         * a present-but-failed decode (corrupt/truncated trailing bytes)
+         * leaves the existing value untouched, matching every other type's
+         * decode-failure fallback in this function.
+         *
+         * pend_reasons is plain calloc()/strdup()-backed throughout
+         * (verified via erase_job_preason()'s FREEUP-only release and every
+         * add_*_preason() constructor in src/lib/preason.c, plus the
+         * capnp-generated decode_HostBasedReason/decode_ResourceBasedReason
+         * using STRING_DUP+calloc), NOT xstring/RTTI-backed like
+         * ckptinfo/rusage's host_name -- so a straightforward
+         * free-and-replace via free_JobPendReason_ptr() is safe here, no
+         * merge-not-replace treatment needed (unlike bug-1690's
+         * WAL_JOB_RUSAGE). */
+        if (wal_type == WAL_JOB_PREASON && len > sizeof(wal_jobstate_t)) {
+            uint8_t present =
+                *((const uint8_t*)payload + sizeof(wal_jobstate_t));
+            xce_job_preason_t* decoded = NULL;
+            if (present && len > sizeof(wal_jobstate_t) + 1) {
+                decoded = decode_jobpreason(
+                    (const uint8_t*)payload + sizeof(wal_jobstate_t) + 1,
+                    len - sizeof(wal_jobstate_t) - 1);
+            }
+            if (present == 0 || decoded != NULL) {
+                if (j->job.pend_reasons != NULL)
+                    free_JobPendReason_ptr(&(j->job.pend_reasons));
+                j->job.pend_reasons = decoded; /* NULL (cleared) or decoded */
+            }
+        }
+
         if (result == JOB_TRANSITION_APPLIED) {
             /* apply_job_update_transition() only carries {jobid, state} on
              * this wire shape (see the doc comment above) and never touches
@@ -7508,6 +7666,19 @@ int persist_job_hook_fun(int type, int situation, int detail,
      * wal_jobstate_t it always was -- zero behavior change for them. */
     uint8_t* ckpt_buf = NULL;
     size_t   ckpt_len = 0;
+    /* bug-1692: WAL_JOB_PREASON's own optional trailing payload -- populated
+     * only by the HOOK_DETAIL_SETPREASON/HOOK_DETAIL_CLRPREASON arm below,
+     * consumed by the shared append at the bottom of this function.
+     * pend_present stays -1 for every OTHER detail (unchanged, plain
+     * wal_jobstate_t payload). Unlike rusage_buf/ckpt_buf above,
+     * pend_reasons has a genuine CLEAR operation (do_clr_preason) that must
+     * be distinguishable on replay from "old-format payload, leave
+     * untouched" -- so this fact type always appends an explicit presence
+     * byte (0 or 1) once pend_present >= 0, even when pend_buf stays NULL
+     * (the clear case). */
+    uint8_t* pend_buf     = NULL;
+    size_t   pend_len     = 0;
+    int      pend_present = -1;
     switch (detail) {
     case HOOK_DETAIL_JOB_START: {
         /* args: cluster_t*, job_pidinfo_t*, job_t* */
@@ -7609,8 +7780,36 @@ int persist_job_hook_fun(int type, int situation, int detail,
         }
         break;
     }
+    case HOOK_DETAIL_SETPREASON:
+    case HOOK_DETAIL_CLRPREASON: {
+        /* args: cluster_t*, job_t* -- same shape as the default arm below,
+         * split out to also capture j->job.pend_reasons (bug-1692: same
+         * silent-field-drop bug class as bug-1678/1687/1689/1690/1691 --
+         * WAL_JOB_PREASON previously carried only {jobid, state}, so a
+         * restart replay left job.pend_reasons permanently NULL/stale even
+         * though set_preason()/do_clr_preason() (job.c) had just set/cleared
+         * it, under the same write lock this AFTER_EVENT fires inside).
+         * pend_present distinguishes "cleared" (0, do_clr_preason has just
+         * FREEUP'd job.pend_reasons back to NULL) from "set" (1, capnp
+         * JobPendReason bytes follow) -- both HOOK_DETAIL_SETPREASON and
+         * HOOK_DETAIL_CLRPREASON share this arm since the shared append
+         * tail below decides purely from j->job.pend_reasons' CURRENT value,
+         * not from which detail fired. */
+        (void)va_arg(ap, cluster_t*);
+        job_t* j = va_arg(ap, job_t*);
+        js.jobid = j->job.jobid; js.state = (uint32_t)j->job.state;
+
+        if (j->job.pend_reasons != NULL) {
+            pend_present = 1;
+            (void)capnp_encode_jobpreason(j->job.pend_reasons, &pend_buf,
+                                          &pend_len);
+        } else {
+            pend_present = 0;
+        }
+        break;
+    }
     default: {
-        /* STOP, RESUME, UPDATE, PREASON — args: cluster_t*, job_t* */
+        /* STOP, RESUME, UPDATE — args: cluster_t*, job_t* */
         (void)va_arg(ap, cluster_t*);
         job_t* j = va_arg(ap, job_t*);
         js.jobid = j->job.jobid; js.state = (uint32_t)j->job.state;
@@ -7656,6 +7855,27 @@ int persist_job_hook_fun(int type, int situation, int detail,
         free(payload);
         return rc;
     }
+    if (pend_present >= 0) {
+        /* bug-1692: append [header][presence byte][optional capnp
+         * JobPendReason bytes] as one contiguous payload -- mirrors the
+         * rusage_buf/ckpt_buf constructions just above (bug-1690/1691), but
+         * ALWAYS appends (even when pend_buf is NULL, the clear case) since
+         * the presence byte itself is the signal both apply seams need to
+         * distinguish "cleared" from "old-format, don't touch". */
+        size_t   total   = sizeof(js) + 1 + pend_len;
+        uint8_t* payload = malloc(total);
+        int      rc;
+        if (!payload) { free(pend_buf); return -ENOMEM; }
+        memcpy(payload, &js, sizeof(js));
+        payload[sizeof(js)] = (uint8_t)pend_present;
+        if (pend_len > 0)
+            memcpy(payload + sizeof(js) + 1, pend_buf, pend_len);
+        free(pend_buf);
+        rc = wal_hook_append_classified(p, wal_type, mode, payload,
+                                        (uint32_t)total);
+        free(payload);
+        return rc;
+    }
     return wal_hook_append_classified(p, wal_type, mode, &js, sizeof(js));
 }
 

src/controller/testsuite/controller_test.c

@@ -88,6 +88,7 @@ extern void test_wal_job_start_fact(void);
 extern void test_wal_job_finish_fact(void);
 extern void test_wal_job_rusage_fact(void);
 extern void test_wal_job_ckpt_fact(void);
+extern void test_wal_job_preason_fact(void);
 extern void test_job_dispatch_fail_count(void);
 extern void test_apply_reapply_idempotency(void);
 extern void test_apply_correctness_verification(void);
@@ -262,6 +263,7 @@ int main(int argc, char* argv[])
     TEST(test_wal_job_finish_fact);
     TEST(test_wal_job_rusage_fact);
     TEST(test_wal_job_ckpt_fact);
+    TEST(test_wal_job_preason_fact);
     TEST(test_job_dispatch_fail_count);
     TEST(test_apply_reapply_idempotency);
     TEST(test_apply_correctness_verification);

src/include/wal_types.h

@@ -187,7 +187,43 @@ typedef enum {
                            * forward state transition -- a checkpoint report
                            * never changes job.state, so gating it the same
                            * way would make it dead code. */
-    WAL_JOB_PREASON = 10,
+    WAL_JOB_PREASON = 10, /* scheduler-reported host/resource-based pending
+                           * reasons (job.pend_reasons, set by set_preason()/
+                           * cleared by do_clr_preason(), job.c). Payload is
+                           * wal_jobstate_t{jobid, state, _pad=0} for any
+                           * entry written before bug-1692. As of bug-1692,
+                           * persist_job_hook_fun's HOOK_DETAIL_SETPREASON/
+                           * HOOK_DETAIL_CLRPREASON arm appends an explicit
+                           * presence byte after the header: 0 = pend_reasons
+                           * was genuinely cleared (adopt NULL on replay), 1
+                           * = trailing Cap'n Proto JobPendReason bytes
+                           * follow (adopt the decoded value). Unlike
+                           * WAL_JOB_CKPT/WAL_JOB_RUSAGE, "trailing bytes
+                           * present or absent" alone cannot distinguish
+                           * "old-format payload" from "field is legitimately
+                           * NULL", because pend_reasons -- uniquely among
+                           * this fact-type family -- has a real CLEAR
+                           * operation (do_clr_preason) that must survive
+                           * replay durably, not just its SET side; the
+                           * presence byte resolves that ambiguity. A payload
+                           * of exactly sizeof(wal_jobstate_t) (no presence
+                           * byte at all) leaves pend_reasons untouched on
+                           * both apply seams, matching the pre-fix
+                           * state-only behavior exactly. Like
+                           * WAL_JOB_RUSAGE/WAL_JOB_CKPT, adoption on the
+                           * apply_committed_entry() seam is NOT gated on a
+                           * forward state transition -- set_preason()/
+                           * do_clr_preason() never change job.state, so
+                           * gating it the same way would make it dead code.
+                           * Adoption is a plain free-and-replace (via the
+                           * capnp-generated free_JobPendReason_ptr()), NOT a
+                           * merge like WAL_JOB_RUSAGE -- job.pend_reasons
+                           * and its nested host/resource reason arrays are
+                           * plain calloc()/strdup() throughout (verified via
+                           * erase_job_preason()'s FREEUP-only release, NOT
+                           * xstring/RTTI-backed like ckptinfo/rusage's
+                           * host_name), so no merge-not-replace hazard
+                           * exists here. */
     WAL_JOB_ARRAY_SUBMIT = 11, /* job-array-single-wal-record: one record for
                                 * the whole array (base spec + index range),
                                 * distinct from WAL_JOB_SUBMIT's single-job