lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): persist a failed job's exit code through WAL_JOB_FAIL (bug-1754)

Browse source 
A worker reports every nonzero job exit as a FAIL (xcecd_job_finished ->
xcecd_notify_job_fail(jobid, stage, exit_code)), not a FINISH. The deferred
WAL_JOB_FAIL fact carried only {jobid, STAGE_FLAG|stage} and no apply seam
touched job.exit_info, so exit_info stayed NULL for every failed job, live
and after replay, and HOOK_DETAIL_JOB_FAIL subscribers saw code 0.

- persist_submit_job_fail_fact(): header + int32_t code trailer, used by
  record_job_fail_fact(). Old entries (no trailer) decode as before.
- apply_job_fail_exit_code(): records the code in exit_info for pre-exit
  stages only; a POST_EXEC code is the script's and leaves exit_info alone.
  Kept out of apply_job_fail_transition() so mark_job_fail()'s internal
  -errno failures (not replayed with a code) stay live/replay consistent.
- Both apply seams adopt the code. apply_wal_entry()'s WAL_JOB_FAIL case
  also stops assigning the wire state 0 for STAGE_FLAG entries and runs the
  real transition, as apply_committed_entry() already does.

New test_wal_job_fail_fact.c (5 tests, 3 RED before the fix); the two
test_worker_resync.c assertions that pinned exit_info == NULL now assert the
worker-reported code. Controller suite: 1228 tests, 0 failures.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-26 03:32:54 +00:00
parent 1ba643b8be
commit 6876e4f6f2
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
10 changed files with 543 additions and 43 deletions
Download patch file Download diff file Expand all files Collapse all files

27
src/controller/job.c
View file

@ -2852,6 +2852,23 @@ job_transition_result_t apply_job_fail_transition(job_t* job,
return JOB_TRANSITION_APPLIED;
}
/* bug-1754: see cluster.h. Kept out of apply_job_fail_transition() on
* purpose -- that core is also driven by mark_job_fail()'s controller-
* internal failures, whose code is a -errno and whose WAL fact (_pad == 0,
* absolute state) never carries it, so recording it there would make the
* live job disagree with its replay. */
void apply_job_fail_exit_code(job_t* job, const job_failinfo_t* failinfo)
{
if (failinfo->stage == JOB_EXEC_STAGE_POST_EXEC
|| failinfo->stage == JOB_EXEC_STAGE_Q_POST_EXEC)
return;
if (job->job.exit_info == NULL)
job->job.exit_info = (job_exitinfo_t*)calloc(1, sizeof(job_exitinfo_t));
if (job->job.exit_info != NULL)
job->job.exit_info->exitcode = failinfo->code;
}
/* Used only by job_ctrl's SIGKILL/SIGTERM not-started-job branch (a client
* command path, do_jobcontrol_op — out of raft-apply-decoupling task 4.1's
* boundary) — mutate-then-notify, unchanged. record_job_fail_fact below is
@ -2907,11 +2924,11 @@ static int record_job_fail_fact(cluster_t* c, job_t* job,
* recomputed shape (0 is written for forward-compatibility/diagnostics
* only); the WAL_JOB_FAIL_STAGE_FLAG bit in `_pad` tells
* apply_committed_entry() to route it to the real
* apply_job_fail_transition() core instead of an absolute overwrite. */
(void)persist_submit_job_fact_ex(&c->persist, WAL_JOB_FAIL,
job->job.jobid, 0,
WAL_JOB_FAIL_STAGE_FLAG |
failinfo->stage);
* apply_job_fail_transition() core instead of an absolute overwrite.
* bug-1754: failinfo->code rides along as a trailer so apply time can
* record the job's exit code (apply_job_fail_exit_code). */
(void)persist_submit_job_fail_fact(&c->persist, job->job.jobid,
failinfo->stage, failinfo->code);
return 0;
}