lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): persist a failed job's finish_time through WAL_JOB_FAIL (bug-1747)

Browse source 
The FAIL half of bug-1747: apply_job_fail_transition() stamps
finish_time = time(NULL) at APPLY time, and WAL_JOB_FAIL carried no time,
so a failed job's finish_time changed on every restart replay / standby
catch-up, and a worker-reported failure found on rejoin could never keep
the worker's own time.

- WAL_JOB_FAIL's trailer grows to int32_t code + uint64_t finish_time
  (seconds). Decoded by length: 16 B (pre-bug-1754), 20 B (code only,
  bug-1754) and 28 B (code + time) all apply as before for what they carry.
- record_job_fail_fact_at(c, job, failinfo, finish_time_ms), the fail-side
  sibling of record_job_finish_fact_at(): 0 = receipt time. on_job_fail()
  uses receipt time; resync's finishedFail branch passes the worker-
  reported finish_time_ms (Req 4.2 now holds for failures too).
- Both apply seams adopt the wire time after the transition.

4 new tests in test_wal_job_fail_fact.c (3 RED before the fix) and the
resync round-trip now asserts the worker-reported failure time (RED
before). Controller suite 1232/0. Factstream HA soak (2 rounds, 60 jobs)
PASS.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-26 09:33:55 +00:00
parent 21e72726f0
commit 1d6c04c28a
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
9 changed files with 349 additions and 68 deletions
Download patch file Download diff file Expand all files Collapse all files

15
src/controller/testsuite/test_worker_resync.c
View file

@ -2838,7 +2838,9 @@ static void test_full_rejoin_reconciliation_round_trip(void)
.jobid = TWR_JOBID_RT_FINFAIL,
.status = JobStateStatus_finishedFail,
.exit_code = worker_reported_exit_code,
.finish_time_ms = 0, .queried = 1
.finish_time_ms = (worker_reported_finish_time_s - 60)
* 1000ULL,
.queried = 1
};
e_unrec = (xce_job_state_entry_t){
.jobid = TWR_JOBID_RT_UNREC,
@ -3013,10 +3015,13 @@ static void test_full_rejoin_reconciliation_round_trip(void)
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_EXIT, j2_finfail->job.state,
"finished-failure: the DEFERRED on_job_fail/record_job_fail_"
"fact fact must apply to EXIT on replay (Req 4.1)");
/* Req 4.2 remains DOCUMENTED-unsatisfiable for the finished-
* FAILURE finish_time specifically (bug-1747) -- not asserted here.
* The worker-reported EXIT CODE, however, is durably recorded via
* WAL_JOB_FAIL's code trailer since bug-1754. */
/* Req 4.2 for the finished-FAILURE outcome: the worker-reported
* exit code (bug-1754) and failure time (bug-1747) both ride
* WAL_JOB_FAIL's trailer and survive restart replay. */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(worker_reported_finish_time_s - 60,
(uint64_t)j2_finfail->job.finish_time,
"bug-1747: the recovered finish_time of a finished-failure must "
"be the WORKER-reported time, not replay-time time(NULL)");
TEST_ASSERT_NOT_NULL_MESSAGE(j2_finfail->job.exit_info,
"bug-1754: a finished-failure's worker-reported exit code must "
"survive restart replay in job.exit_info");