lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): persist a failed job's exit code through WAL_JOB_FAIL (bug-1754)

Browse source 
A worker reports every nonzero job exit as a FAIL (xcecd_job_finished ->
xcecd_notify_job_fail(jobid, stage, exit_code)), not a FINISH. The deferred
WAL_JOB_FAIL fact carried only {jobid, STAGE_FLAG|stage} and no apply seam
touched job.exit_info, so exit_info stayed NULL for every failed job, live
and after replay, and HOOK_DETAIL_JOB_FAIL subscribers saw code 0.

- persist_submit_job_fail_fact(): header + int32_t code trailer, used by
  record_job_fail_fact(). Old entries (no trailer) decode as before.
- apply_job_fail_exit_code(): records the code in exit_info for pre-exit
  stages only; a POST_EXEC code is the script's and leaves exit_info alone.
  Kept out of apply_job_fail_transition() so mark_job_fail()'s internal
  -errno failures (not replayed with a code) stay live/replay consistent.
- Both apply seams adopt the code. apply_wal_entry()'s WAL_JOB_FAIL case
  also stops assigning the wire state 0 for STAGE_FLAG entries and runs the
  real transition, as apply_committed_entry() already does.

New test_wal_job_fail_fact.c (5 tests, 3 RED before the fix); the two
test_worker_resync.c assertions that pinned exit_info == NULL now assert the
worker-reported code. Controller suite: 1228 tests, 0 failures.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-26 03:32:54 +00:00
parent 1ba643b8be
commit 6876e4f6f2
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
10 changed files with 543 additions and 43 deletions
Download patch file Download diff file Expand all files Collapse all files

12
src/include/persist.h
View file

@ -1523,6 +1523,18 @@ int persist_submit_job_start_fact(persist_t* p, uint64_t jobid, int state,
int persist_submit_job_finish_fact(persist_t* p, uint64_t jobid, int state,
uint64_t finish_time, int exit_code);
/* bug-1754: WAL_JOB_FAIL's deferred-apply sibling of
* persist_submit_job_fact_ex() -- header {jobid, 0, WAL_JOB_FAIL_STAGE_FLAG
* | stage} followed by a fixed int32_t code (job_failinfo_t.code). A worker
* reports every nonzero job exit as a FAIL, so for a MAIN-stage failure
* `code` is the job's real exit code; apply_job_fail_exit_code() (job.c)
* adopts it into job.exit_info at both apply seams. A payload of exactly
* sizeof(wal_jobstate_t) (any WAL_JOB_FAIL entry written before this fix)
* carries no code and leaves exit_info untouched. Returns the local-append
* rc, same contract as persist_submit_job_fact(). */
int persist_submit_job_fail_fact(persist_t* p, uint64_t jobid, uint32_t stage,
int code);
/* bug-1679: fire XCE_CONTROLLER_HOOK_JOB's HOOK_DETAIL_JOB_UPDATE AFTER_EVENT
* for a job whose durable WAL fact the caller has ALREADY appended itself
* (e.g. dispatch_a_job_()'s own persist_submit_job_dispatch_fact() call,

8
src/include/wal_types.h
View file

@ -44,9 +44,11 @@ typedef struct {
* as-is". A nonzero _pad's exact interpretation is per-type — see the two
* markers below. */
#define WAL_JOB_FAIL_STAGE_FLAG 0x80000000u
/* WAL_JOB_FAIL: _pad = WAL_JOB_FAIL_STAGE_FLAG | job_failinfo_t.stage.
* Apply time reconstructs a minimal job_failinfo_t from {jobid, stage}
* and calls the real apply_job_fail_transition() against current state. */
/* WAL_JOB_FAIL: _pad = WAL_JOB_FAIL_STAGE_FLAG | job_failinfo_t.stage,
* optionally followed by an int32_t job_failinfo_t.code (bug-1754).
* Apply time reconstructs a minimal job_failinfo_t from {jobid, stage,
* code} and calls the real apply_job_fail_transition() against current
* state, then apply_job_fail_exit_code() when the code is present. */
#define WAL_JOB_UPDATE_POSTDONE_MARK 1u
/* WAL_JOB_UPDATE: _pad = WAL_JOB_UPDATE_POSTDONE_MARK marks a postdone
* merge-fact (`state` is unused/0) — apply time calls