fix(controller): WAL_JOB_PREASON loses job.pend_reasons on replay (bug-1692)

Last of the five field-parity gaps from the original audit (bug-1678,
bug-1687, bug-1689, bug-1690, bug-1691). set_preason()/clear_preason()
set job.pend_reasons (host/resource-based pending-job explanations), but
persist_job_hook_fun's default: arm only ever captured {jobid, state}.
Neither apply seam restored pend_reasons after a restart.

Reuses the ALREADY-EXISTING JobPendReason/HostBasedReason/ResourceBasedReason
capnp structs -- no schema change needed, unlike bug-1690's WAL_JOB_RUSAGE.
Safety-checked against bug-1690's lesson (RTTI/makeString()-backed fields
need merge-not-replace, not free-and-replace): pend_reasons is plain
malloc/strdup-backed throughout (erase_job_preason is FREEUP()-only,
add_host_preason/add_res_preason use strdup()/calloc()), so a
straightforward free-and-replace is safe here, matching bug-1678/1687's
pattern instead.

New wrinkle beyond the four prior fixes: pend_reasons has a genuine CLEAR
operation, and "trailing bytes present or absent" can't distinguish a
legitimate clear from an old-format payload -- added an explicit presence
byte (0=cleared/adopt NULL, 1=capnp bytes follow), verified by two
dedicated "clear survives restart" tests.

11 new tests in test_wal_job_preason_fact.c; RED/GREEN independently
verified (7 genuine pre-fix failures, one test's own wrong premise about
capnp's garbage-byte behavior caught and corrected during RED/GREEN
iteration). Full controller suite: 1178 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake. No ASAN findings.

This closes the field-complete apply parity invariant class the
xce-audit-invariants controller dimension 7 was added for.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 13:14:28 +00:00
parent 1b86e04078
commit d0e5f85aa8
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
7 changed files with 1465 additions and 3 deletions

View file

@ -1067,6 +1067,55 @@ int restore_job_ckpt_state(cluster_t* c, uint64_t jobid, uint32_t state,
return 0;
}
/* bug-1692: WAL_JOB_PREASON's own sibling of restore_job_ckpt_state() above,
* for apply_wal_entry()'s standby catch-up / replicated-apply seam.
*
* has_field distinguishes a pre-fix bare {jobid,state} payload (has_field=0,
* pend_reasons left untouched -- full backward compatibility) from a
* post-fix payload that explicitly carries pend_reasons' new value
* (has_field=1): preason non-NULL adopts a freshly-decoded value, preason
* NULL means the field was genuinely cleared (do_clr_preason) and
* pend_reasons should be reset to NULL too -- pend_reasons is the one
* field-parity fact type in this family (bug-1678/1687/1689/1690/1691) with
* a real CLEAR operation, so unlike restore_job_ckpt_state()/
* restore_job_rusage_state() a NULL decoded value is NOT automatically
* "nothing to adopt".
*
* preason (when non-NULL) is always freshly heap-allocated by
* decode_jobpreason() (persist.c) -- plain calloc()/strdup() throughout
* (verified via erase_job_preason()'s FREEUP-only release and every
* add_*_preason() constructor in src/lib/preason.c, plus the
* capnp-generated decode_HostBasedReason/decode_ResourceBasedReason using
* STRING_DUP+calloc), NOT xstring/RTTI-backed like ckptinfo or rusage's
* host_name -- so a straightforward free-old-then-adopt-new via
* free_JobPendReason_ptr() is safe here, no merge-not-replace treatment
* needed (unlike bug-1690's WAL_JOB_RUSAGE). Ownership of preason always
* transfers to job->job.pend_reasons on success; released here if the job
* is gone. */
int restore_job_preason_state(cluster_t* c, uint64_t jobid, uint32_t state,
int has_field, xce_job_preason_t* preason)
{
job_t* j = NULL;
lock(c->jobmgr, RTTI_LOCK_READ);
key_find(c->jobmgr, (void*)jobid, (void**)&j);
unlock(c->jobmgr);
if (!j) {
if (preason != NULL)
free_JobPendReason_ptr(&preason);
return -ENOENT;
}
lock(j, RTTI_LOCK_WRITE);
j->job.state = (int)state;
if (has_field) {
if (j->job.pend_reasons != NULL)
free_JobPendReason_ptr(&(j->job.pend_reasons));
j->job.pend_reasons = preason; /* NULL (cleared) or the decoded value */
}
unlock(j);
return 0;
}
static int get_nslots_in_alloc(xce_host_alloc_t* alloc)
{
int i;

View file

@ -174,6 +174,7 @@ controller_unit_test_src = [
'testsuite/test_wal_job_finish_fact.c',
'testsuite/test_wal_job_rusage_fact.c',
'testsuite/test_wal_job_ckpt_fact.c',
'testsuite/test_wal_job_preason_fact.c',
'testsuite/test_job_dispatch_fail_count.c',
'testsuite/test_apply_reapply_idempotency.c',
'testsuite/test_apply_correctness_verification.c',

View file

@ -84,6 +84,19 @@ extern int restore_job_rusage_state(cluster_t* c, uint64_t jobid,
* restores it on the primary's own boot replay. */
extern int restore_job_ckpt_state(cluster_t* c, uint64_t jobid,
uint32_t state, char* ckptinfo);
/* restore_job_preason_state defined in job.c (bug-1692): WAL_JOB_PREASON's own
* sibling of restore_job_state() above. Unlike restore_job_ckpt_state()/
* restore_job_rusage_state(), a NULL decoded pointer is not automatically
* "leave untouched" here -- pend_reasons has a genuine CLEAR operation
* (do_clr_preason/HOOK_DETAIL_CLRPREASON) that must durably survive replay
* too, so has_field disambiguates a pre-fix bare {jobid,state} payload
* (has_field=0, pend_reasons left untouched -- full backward compatibility)
* from a post-fix payload that explicitly carries the field's new value
* (has_field=1): preason non-NULL adopts a freshly-decoded value, preason
* NULL means pend_reasons was genuinely cleared and must become NULL too. */
extern int restore_job_preason_state(cluster_t* c, uint64_t jobid,
uint32_t state, int has_field,
xce_job_preason_t* preason);
/* merge_job_runusage defined in job.c (bug-1690): merges a freshly-decoded
* xce_job_runusage_t into *dst rather than freeing-and-replacing it. Needed
* here too (apply_committed_entry's own WAL_JOB_RUSAGE branch below), not
@ -253,6 +266,67 @@ static xce_job_alloc_t* decode_joballoc(const uint8_t* payload, size_t len)
return alloc;
}
/* Serialize xce_job_preason_t to a malloc'd Cap'n Proto buffer (bug-1692):
* lets WAL_JOB_PREASON carry the scheduler's host/resource-based pend
* reasons alongside the {jobid, state} header, so a restart replay can
* reconstruct job.pend_reasons instead of leaving it permanently NULL.
* Mirrors capnp_encode_joballoc above exactly, using the JobPendReason
* struct (already in controller.capnp as the existing Job.pendreason field
* -- no schema change needed here). */
static int capnp_encode_jobpreason(const xce_job_preason_t* pr, uint8_t** out,
size_t* out_len)
{
struct capn c;
struct capn_segment* cs;
struct JobPendReason cjp;
JobPendReason_ptr jpp;
capn_init_malloc(&c);
cs = capn_root(&c).seg;
encode_JobPendReason(cs, &cjp, (xce_job_preason_t*)pr);
jpp = new_JobPendReason(cs);
write_JobPendReason(&cjp, jpp);
capn_setp(capn_root(&c), 0, jpp.p);
*out_len = capn_size(&c);
*out = malloc(*out_len);
if (!*out) { capn_free(&c); return -ENOMEM; }
capn_write_mem(&c, *out, *out_len, 0);
capn_free(&c);
return 0;
}
/* Decode a Cap'n Proto JobPendReason buffer (bug-1692) into a freshly heap-
* allocated xce_job_preason_t*, mirroring decode_joballoc()'s pattern above.
* Returns NULL on a NULL/empty payload; caller owns the result. The decoded
* tree is plain calloc()/strdup() throughout (capnp-generated
* decode_HostBasedReason/decode_ResourceBasedReason use STRING_DUP + calloc,
* verified in controller.capnp.c), NOT xstring/RTTI-backed like ckptinfo or
* rusage's host_name -- release via free_JobPendReason_ptr (plain free()),
* never dereference(). */
static xce_job_preason_t* decode_jobpreason(const uint8_t* payload,
size_t len)
{
struct capn c_ctx;
JobPendReason_ptr jpp;
struct JobPendReason cjp;
xce_job_preason_t* preason;
if (!payload || len == 0)
return NULL;
capn_init_mem(&c_ctx, payload, len, 0);
jpp.p = capn_getp(capn_root(&c_ctx), 0, 1);
memset(&cjp, 0, sizeof(cjp));
read_JobPendReason(&cjp, jpp);
preason = calloc(1, sizeof(*preason));
if (!preason) { capn_free(&c_ctx); return NULL; }
decode_JobPendReason(preason, &cjp);
capn_free(&c_ctx);
return preason;
}
/* Serialize job_pidinfo_t to a malloc'd Cap'n Proto buffer (bug-1687):
* lets WAL_JOB_START carry the worker-reported pids/pgids/exdata alongside
* the {jobid, state} header, so a restart replay can reconstruct
@ -3746,7 +3820,6 @@ int apply_wal_entry(persist_t* p, uint32_t type, uint64_t lsn,
case WAL_JOB_STOP:
case WAL_JOB_RESUME:
case WAL_JOB_UPDATE:
case WAL_JOB_PREASON:
/* Payload: wal_jobstate_t {jobid, state} */
if (payload_len >= sizeof(wal_jobstate_t)) {
const wal_jobstate_t* js = (const wal_jobstate_t*)payload;
@ -3754,6 +3827,46 @@ int apply_wal_entry(persist_t* p, uint32_t type, uint64_t lsn,
}
break;
case WAL_JOB_PREASON: {
/* bug-1692: split out of the shared group above -- mirrors
* WAL_JOB_CKPT's/WAL_JOB_RUSAGE's splits (bug-1690/1691). Unlike
* those, pend_reasons has a genuine "clear" operation
* (do_clr_preason), so the payload beyond the wal_jobstate_t header
* always starts with an explicit presence byte (0 = cleared, adopt
* NULL; 1 = capnp JobPendReason bytes follow, adopt the decoded
* value) on any entry written after this fix. A payload of exactly
* sizeof(wal_jobstate_t) (no presence byte -- any WAL_JOB_PREASON
* entry written before this fix) leaves pend_reasons untouched,
* matching the pre-fix state-only behavior exactly. */
if (payload_len >= sizeof(wal_jobstate_t)) {
const wal_jobstate_t* js = (const wal_jobstate_t*)payload;
int has_field = 0;
xce_job_preason_t* preason = NULL;
if (payload_len > sizeof(wal_jobstate_t)) {
uint8_t present =
*((const uint8_t*)payload + sizeof(wal_jobstate_t));
if (present) {
if (payload_len > sizeof(wal_jobstate_t) + 1) {
preason = decode_jobpreason(
(const uint8_t*)payload + sizeof(wal_jobstate_t)
+ 1,
payload_len - sizeof(wal_jobstate_t) - 1);
}
/* present-but-failed-decode (corrupt/truncated trailing
* bytes) must NOT be treated as a clear -- leave
* pend_reasons untouched, matching every other decode
* helper's failure fallback in this function. */
has_field = (preason != NULL);
} else {
has_field = 1; /* explicit clear -- adopt NULL */
}
}
restore_job_preason_state(p->cluster, js->jobid, js->state,
has_field, preason);
}
break;
}
case WAL_JOB_CKPT: {
/* bug-1691: split out of the shared group above -- mirrors
* WAL_JOB_RUSAGE's split just below (bug-1690). This is the one
@ -4258,6 +4371,51 @@ int apply_committed_entry(persist_t* p, uint32_t wal_type, uint64_t lsn,
}
}
/* bug-1692: WAL_JOB_PREASON mirrors WAL_JOB_CKPT's/WAL_JOB_RUSAGE's
* unconditional-adopt pattern immediately above -- set_preason()/
* do_clr_preason() (job.c) never touch job.state, so gating
* adoption on result==JOB_TRANSITION_APPLIED would make it dead
* code: js->state is always the job's unchanged current state
* (persist_job_hook_fun's HOOK_DETAIL_SETPREASON/CLRPREASON arm
* captures it verbatim), so apply_job_update_transition() always
* reports JOB_TRANSITION_NOOP_DUPLICATE here. Decode/adopt
* unconditionally, before the transition-guarded block below.
*
* Unlike CKPT/RUSAGE, a presence byte (not "trailing bytes exist or
* not") disambiguates a genuine CLEAR (do_clr_preason) from an
* old-format payload -- see decode_jobpreason()'s/
* restore_job_preason_state()'s doc comments and WAL_JOB_PREASON's
* own doc comment in wal_types.h for the full rationale. `present ==
* 0` (explicit clear) or a successful decode both trigger adoption;
* a present-but-failed decode (corrupt/truncated trailing bytes)
* leaves the existing value untouched, matching every other type's
* decode-failure fallback in this function.
*
* pend_reasons is plain calloc()/strdup()-backed throughout
* (verified via erase_job_preason()'s FREEUP-only release and every
* add_*_preason() constructor in src/lib/preason.c, plus the
* capnp-generated decode_HostBasedReason/decode_ResourceBasedReason
* using STRING_DUP+calloc), NOT xstring/RTTI-backed like
* ckptinfo/rusage's host_name -- so a straightforward
* free-and-replace via free_JobPendReason_ptr() is safe here, no
* merge-not-replace treatment needed (unlike bug-1690's
* WAL_JOB_RUSAGE). */
if (wal_type == WAL_JOB_PREASON && len > sizeof(wal_jobstate_t)) {
uint8_t present =
*((const uint8_t*)payload + sizeof(wal_jobstate_t));
xce_job_preason_t* decoded = NULL;
if (present && len > sizeof(wal_jobstate_t) + 1) {
decoded = decode_jobpreason(
(const uint8_t*)payload + sizeof(wal_jobstate_t) + 1,
len - sizeof(wal_jobstate_t) - 1);
}
if (present == 0 || decoded != NULL) {
if (j->job.pend_reasons != NULL)
free_JobPendReason_ptr(&(j->job.pend_reasons));
j->job.pend_reasons = decoded; /* NULL (cleared) or decoded */
}
}
if (result == JOB_TRANSITION_APPLIED) {
/* apply_job_update_transition() only carries {jobid, state} on
* this wire shape (see the doc comment above) and never touches
@ -7508,6 +7666,19 @@ int persist_job_hook_fun(int type, int situation, int detail,
* wal_jobstate_t it always was -- zero behavior change for them. */
uint8_t* ckpt_buf = NULL;
size_t ckpt_len = 0;
/* bug-1692: WAL_JOB_PREASON's own optional trailing payload -- populated
* only by the HOOK_DETAIL_SETPREASON/HOOK_DETAIL_CLRPREASON arm below,
* consumed by the shared append at the bottom of this function.
* pend_present stays -1 for every OTHER detail (unchanged, plain
* wal_jobstate_t payload). Unlike rusage_buf/ckpt_buf above,
* pend_reasons has a genuine CLEAR operation (do_clr_preason) that must
* be distinguishable on replay from "old-format payload, leave
* untouched" -- so this fact type always appends an explicit presence
* byte (0 or 1) once pend_present >= 0, even when pend_buf stays NULL
* (the clear case). */
uint8_t* pend_buf = NULL;
size_t pend_len = 0;
int pend_present = -1;
switch (detail) {
case HOOK_DETAIL_JOB_START: {
/* args: cluster_t*, job_pidinfo_t*, job_t* */
@ -7609,8 +7780,36 @@ int persist_job_hook_fun(int type, int situation, int detail,
}
break;
}
case HOOK_DETAIL_SETPREASON:
case HOOK_DETAIL_CLRPREASON: {
/* args: cluster_t*, job_t* -- same shape as the default arm below,
* split out to also capture j->job.pend_reasons (bug-1692: same
* silent-field-drop bug class as bug-1678/1687/1689/1690/1691 --
* WAL_JOB_PREASON previously carried only {jobid, state}, so a
* restart replay left job.pend_reasons permanently NULL/stale even
* though set_preason()/do_clr_preason() (job.c) had just set/cleared
* it, under the same write lock this AFTER_EVENT fires inside).
* pend_present distinguishes "cleared" (0, do_clr_preason has just
* FREEUP'd job.pend_reasons back to NULL) from "set" (1, capnp
* JobPendReason bytes follow) -- both HOOK_DETAIL_SETPREASON and
* HOOK_DETAIL_CLRPREASON share this arm since the shared append
* tail below decides purely from j->job.pend_reasons' CURRENT value,
* not from which detail fired. */
(void)va_arg(ap, cluster_t*);
job_t* j = va_arg(ap, job_t*);
js.jobid = j->job.jobid; js.state = (uint32_t)j->job.state;
if (j->job.pend_reasons != NULL) {
pend_present = 1;
(void)capnp_encode_jobpreason(j->job.pend_reasons, &pend_buf,
&pend_len);
} else {
pend_present = 0;
}
break;
}
default: {
/* STOP, RESUME, UPDATE, PREASON — args: cluster_t*, job_t* */
/* STOP, RESUME, UPDATE — args: cluster_t*, job_t* */
(void)va_arg(ap, cluster_t*);
job_t* j = va_arg(ap, job_t*);
js.jobid = j->job.jobid; js.state = (uint32_t)j->job.state;
@ -7656,6 +7855,27 @@ int persist_job_hook_fun(int type, int situation, int detail,
free(payload);
return rc;
}
if (pend_present >= 0) {
/* bug-1692: append [header][presence byte][optional capnp
* JobPendReason bytes] as one contiguous payload -- mirrors the
* rusage_buf/ckpt_buf constructions just above (bug-1690/1691), but
* ALWAYS appends (even when pend_buf is NULL, the clear case) since
* the presence byte itself is the signal both apply seams need to
* distinguish "cleared" from "old-format, don't touch". */
size_t total = sizeof(js) + 1 + pend_len;
uint8_t* payload = malloc(total);
int rc;
if (!payload) { free(pend_buf); return -ENOMEM; }
memcpy(payload, &js, sizeof(js));
payload[sizeof(js)] = (uint8_t)pend_present;
if (pend_len > 0)
memcpy(payload + sizeof(js) + 1, pend_buf, pend_len);
free(pend_buf);
rc = wal_hook_append_classified(p, wal_type, mode, payload,
(uint32_t)total);
free(payload);
return rc;
}
return wal_hook_append_classified(p, wal_type, mode, &js, sizeof(js));
}

View file

@ -88,6 +88,7 @@ extern void test_wal_job_start_fact(void);
extern void test_wal_job_finish_fact(void);
extern void test_wal_job_rusage_fact(void);
extern void test_wal_job_ckpt_fact(void);
extern void test_wal_job_preason_fact(void);
extern void test_job_dispatch_fail_count(void);
extern void test_apply_reapply_idempotency(void);
extern void test_apply_correctness_verification(void);
@ -262,6 +263,7 @@ int main(int argc, char* argv[])
TEST(test_wal_job_finish_fact);
TEST(test_wal_job_rusage_fact);
TEST(test_wal_job_ckpt_fact);
TEST(test_wal_job_preason_fact);
TEST(test_job_dispatch_fail_count);
TEST(test_apply_reapply_idempotency);
TEST(test_apply_correctness_verification);

File diff suppressed because it is too large Load diff

View file

@ -187,7 +187,43 @@ typedef enum {
* forward state transition -- a checkpoint report
* never changes job.state, so gating it the same
* way would make it dead code. */
WAL_JOB_PREASON = 10,
WAL_JOB_PREASON = 10, /* scheduler-reported host/resource-based pending
* reasons (job.pend_reasons, set by set_preason()/
* cleared by do_clr_preason(), job.c). Payload is
* wal_jobstate_t{jobid, state, _pad=0} for any
* entry written before bug-1692. As of bug-1692,
* persist_job_hook_fun's HOOK_DETAIL_SETPREASON/
* HOOK_DETAIL_CLRPREASON arm appends an explicit
* presence byte after the header: 0 = pend_reasons
* was genuinely cleared (adopt NULL on replay), 1
* = trailing Cap'n Proto JobPendReason bytes
* follow (adopt the decoded value). Unlike
* WAL_JOB_CKPT/WAL_JOB_RUSAGE, "trailing bytes
* present or absent" alone cannot distinguish
* "old-format payload" from "field is legitimately
* NULL", because pend_reasons -- uniquely among
* this fact-type family -- has a real CLEAR
* operation (do_clr_preason) that must survive
* replay durably, not just its SET side; the
* presence byte resolves that ambiguity. A payload
* of exactly sizeof(wal_jobstate_t) (no presence
* byte at all) leaves pend_reasons untouched on
* both apply seams, matching the pre-fix
* state-only behavior exactly. Like
* WAL_JOB_RUSAGE/WAL_JOB_CKPT, adoption on the
* apply_committed_entry() seam is NOT gated on a
* forward state transition -- set_preason()/
* do_clr_preason() never change job.state, so
* gating it the same way would make it dead code.
* Adoption is a plain free-and-replace (via the
* capnp-generated free_JobPendReason_ptr()), NOT a
* merge like WAL_JOB_RUSAGE -- job.pend_reasons
* and its nested host/resource reason arrays are
* plain calloc()/strdup() throughout (verified via
* erase_job_preason()'s FREEUP-only release, NOT
* xstring/RTTI-backed like ckptinfo/rusage's
* host_name), so no merge-not-replace hazard
* exists here. */
WAL_JOB_ARRAY_SUBMIT = 11, /* job-array-single-wal-record: one record for
* the whole array (base spec + index range),
* distinct from WAL_JOB_SUBMIT's single-job