lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): persist a failed job's finish_time through WAL_JOB_FAIL (bug-1747)

Browse source 
The FAIL half of bug-1747: apply_job_fail_transition() stamps
finish_time = time(NULL) at APPLY time, and WAL_JOB_FAIL carried no time,
so a failed job's finish_time changed on every restart replay / standby
catch-up, and a worker-reported failure found on rejoin could never keep
the worker's own time.

- WAL_JOB_FAIL's trailer grows to int32_t code + uint64_t finish_time
  (seconds). Decoded by length: 16 B (pre-bug-1754), 20 B (code only,
  bug-1754) and 28 B (code + time) all apply as before for what they carry.
- record_job_fail_fact_at(c, job, failinfo, finish_time_ms), the fail-side
  sibling of record_job_finish_fact_at(): 0 = receipt time. on_job_fail()
  uses receipt time; resync's finishedFail branch passes the worker-
  reported finish_time_ms (Req 4.2 now holds for failures too).
- Both apply seams adopt the wire time after the transition.

4 new tests in test_wal_job_fail_fact.c (3 RED before the fix) and the
resync round-trip now asserts the worker-reported failure time (RED
before). Controller suite 1232/0. Factstream HA soak (2 rounds, 60 jobs)
PASS.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-26 09:33:55 +00:00
parent 21e72726f0
commit 1d6c04c28a
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
9 changed files with 349 additions and 68 deletions
Download patch file Download diff file Expand all files Collapse all files

19
src/include/persist.h
View file

@ -1523,17 +1523,18 @@ int persist_submit_job_start_fact(persist_t* p, uint64_t jobid, int state,
int persist_submit_job_finish_fact(persist_t* p, uint64_t jobid, int state,
uint64_t finish_time, int exit_code);
/* bug-1754: WAL_JOB_FAIL's deferred-apply sibling of
/* bug-1754/bug-1747: WAL_JOB_FAIL's deferred-apply sibling of
* persist_submit_job_fact_ex() -- header {jobid, 0, WAL_JOB_FAIL_STAGE_FLAG
* | stage} followed by a fixed int32_t code (job_failinfo_t.code). A worker
* reports every nonzero job exit as a FAIL, so for a MAIN-stage failure
* `code` is the job's real exit code; apply_job_fail_exit_code() (job.c)
* adopts it into job.exit_info at both apply seams. A payload of exactly
* sizeof(wal_jobstate_t) (any WAL_JOB_FAIL entry written before this fix)
* carries no code and leaves exit_info untouched. Returns the local-append
* rc, same contract as persist_submit_job_fact(). */
* | stage} followed by a fixed int32_t code (job_failinfo_t.code) and a
* fixed uint64_t finish_time (seconds). A worker reports every nonzero job
* exit as a FAIL, so for a MAIN-stage failure `code` is the job's real exit
* code; apply_job_fail_exit_code() (job.c) adopts it into job.exit_info and
* both apply seams adopt finish_time instead of apply-time time(NULL).
* Older payloads decode by length: exactly sizeof(wal_jobstate_t) (before
* bug-1754) carries neither; header + code (bug-1754) carries no time.
* Returns the local-append rc, same contract as persist_submit_job_fact(). */
int persist_submit_job_fail_fact(persist_t* p, uint64_t jobid, uint32_t stage,
int code);
int code, uint64_t finish_time);
/* bug-1679: fire XCE_CONTROLLER_HOOK_JOB's HOOK_DETAIL_JOB_UPDATE AFTER_EVENT
* for a job whose durable WAL fact the caller has ALREADY appended itself