fix(controller): persist a failed job's exit code through WAL_JOB_FAIL (bug-1754)

A worker reports every nonzero job exit as a FAIL (xcecd_job_finished ->
xcecd_notify_job_fail(jobid, stage, exit_code)), not a FINISH. The deferred
WAL_JOB_FAIL fact carried only {jobid, STAGE_FLAG|stage} and no apply seam
touched job.exit_info, so exit_info stayed NULL for every failed job, live
and after replay, and HOOK_DETAIL_JOB_FAIL subscribers saw code 0.

- persist_submit_job_fail_fact(): header + int32_t code trailer, used by
  record_job_fail_fact(). Old entries (no trailer) decode as before.
- apply_job_fail_exit_code(): records the code in exit_info for pre-exit
  stages only; a POST_EXEC code is the script's and leaves exit_info alone.
  Kept out of apply_job_fail_transition() so mark_job_fail()'s internal
  -errno failures (not replayed with a code) stay live/replay consistent.
- Both apply seams adopt the code. apply_wal_entry()'s WAL_JOB_FAIL case
  also stops assigning the wire state 0 for STAGE_FLAG entries and runs the
  real transition, as apply_committed_entry() already does.

New test_wal_job_fail_fact.c (5 tests, 3 RED before the fix); the two
test_worker_resync.c assertions that pinned exit_info == NULL now assert the
worker-reported code. Controller suite: 1228 tests, 0 failures.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-26 03:32:54 +00:00
parent 1ba643b8be
commit 6876e4f6f2
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
10 changed files with 543 additions and 43 deletions

View file

@ -3013,31 +3013,15 @@ static void test_full_rejoin_reconciliation_round_trip(void)
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_EXIT, j2_finfail->job.state,
"finished-failure: the DEFERRED on_job_fail/record_job_fail_"
"fact fact must apply to EXIT on replay (Req 4.1)");
/* Req 4.2 is DOCUMENTED-unsatisfiable for the finished-FAILURE
* finish_time specifically (bug-1747, pre-existing, systemic,
* explicitly deferred, unrelated to this task's own boundary) --
* NOT re-asserted here as if it were correct. This round-trip test
* additionally confirms, and CONCERNS reports, a related, more
* complete gap bug-1747's own text does not mention: the worker-
* reported EXIT CODE (entry->exit_code, plumbed all the way into
* job_failinfo_t.code by resync_apply_finished_entry, resync.c) is
* not merely mistimed but is NEVER durably recorded or applied to
* job.job.exit_info AT ALL for a FAIL outcome, by any existing path
* in this codebase -- record_job_fail_fact's own WAL_JOB_FAIL
* payload (persist_submit_job_fact_ex(..., WAL_JOB_FAIL, jobid, 0,
* WAL_JOB_FAIL_STAGE_FLAG|stage), job.c) carries no code slot at
* all, and apply_job_fail_transition() (job.c) never touches job.
* job.exit_info. This is asserted honestly below as NULL -- the
* TRUE current behavior -- rather than asserting a "correct" value
* this codebase cannot actually produce (this file's own task 3.2
* Implementation Notes: "a green suite is not proof of correctness
* when the test itself encodes the wrong expectation"). */
TEST_ASSERT_NULL_MESSAGE(j2_finfail->job.exit_info,
"pre-existing, systemic gap (see comment above, reported in "
"this task's own CONCERNS, not fixed here): a worker-reported "
"exit code for a FAIL outcome is never durably recorded by any "
"existing path in this codebase -- exit_info stays NULL, not "
"the worker-reported code");
/* Req 4.2 remains DOCUMENTED-unsatisfiable for the finished-
* FAILURE finish_time specifically (bug-1747) -- not asserted here.
* The worker-reported EXIT CODE, however, is durably recorded via
* WAL_JOB_FAIL's code trailer since bug-1754. */
TEST_ASSERT_NOT_NULL_MESSAGE(j2_finfail->job.exit_info,
"bug-1754: a finished-failure's worker-reported exit code must "
"survive restart replay in job.exit_info");
TEST_ASSERT_EQUAL_INT(worker_reported_exit_code,
j2_finfail->job.exit_info->exitcode);
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_PEND, j2_unrec->job.state,
"unrecognized: the EAGER record_job_reconcile_pending_fact's "
@ -4080,12 +4064,11 @@ static void test_burst_recovery_unbounded_by_job_count(void)
"EXIT, not DONE, distinctly from its 5 finished-success "
"siblings in the SAME round");
/* Req 4.2 remains documented-unsatisfiable for FAIL's own
* finish_time specifically (bug-1747, pre-existing, systemic, out
* of this task's boundary -- see tasks 7.1/7.2's own identical
* finding); exit_info stays NULL for the same pre-existing reason,
* asserted honestly rather than as a wrong "correct" value this
* codebase cannot actually produce. */
TEST_ASSERT_NULL(j2_lost[TWR_BURST_N_LOST - 1]->job.exit_info);
* finish_time specifically (bug-1747). Its exit code is recorded
* since bug-1754. */
TEST_ASSERT_NOT_NULL(j2_lost[TWR_BURST_N_LOST - 1]->job.exit_info);
TEST_ASSERT_EQUAL_INT(lost_fail_exit_code,
j2_lost[TWR_BURST_N_LOST - 1]->job.exit_info->exitcode);
/* The 'none duplicated' completion criterion, confirmed to
* survive a restart replay too: the already-DONE job's ORIGINAL