lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): persist a failed job's finish_time through WAL_JOB_FAIL (bug-1747)

Browse source 
The FAIL half of bug-1747: apply_job_fail_transition() stamps
finish_time = time(NULL) at APPLY time, and WAL_JOB_FAIL carried no time,
so a failed job's finish_time changed on every restart replay / standby
catch-up, and a worker-reported failure found on rejoin could never keep
the worker's own time.

- WAL_JOB_FAIL's trailer grows to int32_t code + uint64_t finish_time
  (seconds). Decoded by length: 16 B (pre-bug-1754), 20 B (code only,
  bug-1754) and 28 B (code + time) all apply as before for what they carry.
- record_job_fail_fact_at(c, job, failinfo, finish_time_ms), the fail-side
  sibling of record_job_finish_fact_at(): 0 = receipt time. on_job_fail()
  uses receipt time; resync's finishedFail branch passes the worker-
  reported finish_time_ms (Req 4.2 now holds for failures too).
- Both apply seams adopt the wire time after the transition.

4 new tests in test_wal_job_fail_fact.c (3 RED before the fix) and the
resync round-trip now asserts the worker-reported failure time (RED
before). Controller suite 1232/0. Factstream HA soak (2 rounds, 60 jobs)
PASS.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-26 09:33:55 +00:00
parent 21e72726f0
commit 1d6c04c28a
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
9 changed files with 349 additions and 68 deletions
Download patch file Download diff file Expand all files Collapse all files

39
src/controller/resync.c
View file

@ -639,18 +639,12 @@ static void resync_apply_running_entry(cluster_t* c, worker_t* w,
* the job table is a silent no-op, mirroring on_job_finish()'s own
* "can not find job" behavior for an unknown jobid.
*
* finishedFail: still routed through the existing, UNMODIFIED on_job_fail()
* /record_job_fail_fact() -- Requirement 4.2 remains unsatisfiable for this
* outcome via any existing path in this codebase, not a reconciliation-
* specific gap: WAL_JOB_FAIL's payload (persist_submit_job_fail_fact():
* stage plus, since bug-1754, the exit code) carries no finish-time slot, and apply_job_fail_transition() (job.c)
* unconditionally sets job->job.finish_time = time(NULL) at APPLY time
* (not even receipt time) for EVERY fail-reporting caller in this system,
* confirmed by direct inspection -- a pre-existing, systemic gap that
* predates this feature and is out of this task's boundary to fix (see
* buglog.json bug-1747, re-scoped). The reconciled job's STATE (EXIT) is
* always correct either way; only its recorded finish_time reflects
* apply-processing time rather than the worker's true report. */
* finishedFail: routed through record_job_fail_fact_at() (bug-1747), the
* fail-side sibling of record_job_finish_fact_at(): the worker-reported
* finish_time_ms (receipt time when 0) and exit code (bug-1754) both ride
* WAL_JOB_FAIL's trailer, so Req 4.2 now holds for this outcome too. Same
* lookup/lock/unlock/dereference sequence as the finishedOk branch; an
* unknown jobid is a silent no-op. */
static void resync_apply_finished_entry(cluster_t* c,
xce_job_state_entry_t* entry)
{
@ -669,12 +663,21 @@ static void resync_apply_finished_entry(cluster_t* c,
dereference(job);
}
} else {
job_failinfo_t fail
= { .jobid = entry->jobid,
.stage = JOB_EXEC_STAGE_MAIN,
.code = entry->exit_code,
.detail = (char*)"worker-rejoin-resync reconciliation" };
on_job_fail(c, &fail);
job_t* job = resync_lookup_job(c, entry->jobid);
if (job != NULL) {
job_failinfo_t fail
= { .jobid = entry->jobid,
.stage = JOB_EXEC_STAGE_MAIN,
.code = entry->exit_code,
.detail = (char*)"worker-rejoin-resync reconciliation" };
lock(job, RTTI_LOCK_WRITE);
record_job_fail_fact_at(c, job, &fail, entry->finish_time_ms);
unlock(job);
dereference(job);
}
}
}