lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): persist a failed job's exit code through WAL_JOB_FAIL (bug-1754)

Browse source 
A worker reports every nonzero job exit as a FAIL (xcecd_job_finished ->
xcecd_notify_job_fail(jobid, stage, exit_code)), not a FINISH. The deferred
WAL_JOB_FAIL fact carried only {jobid, STAGE_FLAG|stage} and no apply seam
touched job.exit_info, so exit_info stayed NULL for every failed job, live
and after replay, and HOOK_DETAIL_JOB_FAIL subscribers saw code 0.

- persist_submit_job_fail_fact(): header + int32_t code trailer, used by
  record_job_fail_fact(). Old entries (no trailer) decode as before.
- apply_job_fail_exit_code(): records the code in exit_info for pre-exit
  stages only; a POST_EXEC code is the script's and leaves exit_info alone.
  Kept out of apply_job_fail_transition() so mark_job_fail()'s internal
  -errno failures (not replayed with a code) stay live/replay consistent.
- Both apply seams adopt the code. apply_wal_entry()'s WAL_JOB_FAIL case
  also stops assigning the wire state 0 for STAGE_FLAG entries and runs the
  real transition, as apply_committed_entry() already does.

New test_wal_job_fail_fact.c (5 tests, 3 RED before the fix); the two
test_worker_resync.c assertions that pinned exit_info == NULL now assert the
worker-reported code. Controller suite: 1228 tests, 0 failures.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-26 03:32:54 +00:00
parent 1ba643b8be
commit 6876e4f6f2
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
10 changed files with 543 additions and 43 deletions
Download patch file Download diff file Expand all files Collapse all files

72
src/controller/persist.c
View file

@ -4113,6 +4113,46 @@ int apply_wal_entry(persist_t* p, uint32_t type, uint64_t lsn,
break;
case WAL_JOB_FAIL:
/* bug-1754: a deferred fail-fact (WAL_JOB_FAIL_STAGE_FLAG) carries
* state 0 on the wire -- recompute via the real transition core,
* mirroring apply_committed_entry(), and adopt the optional int32_t
* code trailer into exit_info. _pad == 0 is the legacy absolute-
* state shape and falls through to restore_job_state(). */
if (payload_len >= sizeof(wal_jobstate_t)
&& (((const wal_jobstate_t*)payload)->_pad
& WAL_JOB_FAIL_STAGE_FLAG)) {
const wal_jobstate_t* js = (const wal_jobstate_t*)payload;
int32_t wire_code = 0;
bool have_code = payload_len
>= sizeof(wal_jobstate_t) + sizeof(wire_code);
job_t* j = NULL;
if (have_code)
memcpy(&wire_code,
(const uint8_t*)payload + sizeof(wal_jobstate_t),
sizeof(wire_code));
lock(p->cluster->jobmgr, RTTI_LOCK_READ);
key_find(p->cluster->jobmgr, (void*)js->jobid, (void**)&j);
unlock(p->cluster->jobmgr);
if (j != NULL) {
job_failinfo_t failinfo = {
.jobid = js->jobid,
.stage = js->_pad & ~WAL_JOB_FAIL_STAGE_FLAG,
.code = (int)wire_code,
.detail = NULL,
};
lock(j, RTTI_LOCK_WRITE);
if (apply_job_fail_transition(j, &failinfo)
== JOB_TRANSITION_APPLIED
&& have_code)
apply_job_fail_exit_code(j, &failinfo);
unlock(j);
}
break;
}
/* fall through */
case WAL_JOB_STOP:
case WAL_JOB_RESUME:
case WAL_JOB_UPDATE:
@ -5006,14 +5046,27 @@ int apply_committed_entry(persist_t* p, uint32_t wal_type, uint64_t lsn,
job_failinfo_t failinfo;
bool have_failinfo = false;
if (js->_pad & WAL_JOB_FAIL_STAGE_FLAG) {
int32_t wire_code = 0;
bool have_code = len >= sizeof(wal_jobstate_t)
+ sizeof(wire_code);
/* bug-1754: persist_submit_job_fail_fact()'s int32_t code
* trailer; absent on entries written before the fix. */
if (have_code)
memcpy(&wire_code,
(const uint8_t*)payload + sizeof(wal_jobstate_t),
sizeof(wire_code));
failinfo = (job_failinfo_t){
.jobid = js->jobid,
.stage = js->_pad & ~WAL_JOB_FAIL_STAGE_FLAG,
.code = 0,
.code = (int)wire_code,
.detail = NULL,
};
have_failinfo = true;
result = apply_job_fail_transition(j, &failinfo);
if (have_code && result == JOB_TRANSITION_APPLIED)
apply_job_fail_exit_code(j, &failinfo);
} else {
result = apply_job_update_transition(j, (int)js->state);
}
@ -8062,6 +8115,23 @@ int persist_submit_job_finish_fact(persist_t* p, uint64_t jobid, int state,
payload, (uint32_t)sizeof(payload));
}
/* bug-1754: see persist.h's doc comment. Same native fixed-size trailer
* style as persist_submit_job_finish_fact() above. */
int persist_submit_job_fail_fact(persist_t* p, uint64_t jobid, uint32_t stage,
int code)
{
wal_jobstate_t js = { .jobid = jobid, .state = 0,
._pad = WAL_JOB_FAIL_STAGE_FLAG | stage };
int32_t c32 = (int32_t)code;
uint8_t payload[sizeof(js) + sizeof(c32)];
memcpy(payload, &js, sizeof(js));
memcpy(payload + sizeof(js), &c32, sizeof(c32));
return wal_hook_append_classified(p, WAL_JOB_FAIL, WAL_MODE_RETAIN,
payload, (uint32_t)sizeof(payload));
}
/* bug-1679: see persist.h's doc comment for the full rationale. Mirrors
* apply_committed_entry()'s own identical Hazard-1-guarded hook fire
* (its shared WAL_JOB_START/.../DISPATCH case, right after its