---
id: "bug-1747"
status: "fixed"
status_updated: "2026-09-26"
status_note: "FINISH half eec2596c; FAIL half 1d6c04c2: WAL_JOB_FAIL trailer carries finish_time, record_job_fail_fact_at() for resync."
component: "controller"
severity: "unknown"
timestamp: "2026-09-01T00:00:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/job.c"]
tags: ["worker-rejoin-resync", "task-3.2", "resync-c", "job-c", "requirement-4.2", "finish-time", "wal-job-fail", "wal-job-finish", "apply_job_fail_transition", "pre-existing", "predates-feature", "systemic", "resync_apply_report", "record_job_finish_fact_at"]
related: ["bug-1689", "bug-1678", "bug-1746", "bug-1631"]
fix_commits: ["eec2596c", "1d6c04c2"]
legacy: true
---

# bug-1747: systemic gap predating worker-rejoin-resync entirely, discovered while implementing task 3.2…

## Description
Pre-existing, systemic gap predating worker-rejoin-resync entirely, discovered while implementing task 3.2 (resync_apply_report, resync.c): WAL_JOB_FAIL carries no fail-time/finish-time slot in its wire payload at ANY layer, and apply_job_fail_transition() (job.c) unconditionally stamps job.finish_time = time(NULL) at APPLY time (not even receipt time) for EVERY fail-reporting caller in this codebase -- not just reconciliation. This means Requirement 4.2 ('the Controller shall record the finish time the Worker reported ... rather than the time the Controller processed the reconciliation') is currently unsatisfiable for a reconciled finished-FAILURE outcome via any existing path. UPDATE (2026-09-01, coordinator-directed re-scope): the sibling FINISH-success half of this gap is FIXED -- see 'fix' field below; only the FAIL half remains open.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
job_failinfo_t (xce/types.h) is {jobid, stage, code, detail} -- no finish_time field, and WAL_JOB_FAIL's durable payload (record_job_fail_fact(), job.c: persist_submit_job_fact_ex(&c->persist, WAL_JOB_FAIL, jobid, 0, WAL_JOB_FAIL_STAGE_FLAG|stage)) carries no finish-time slot at all -- confirmed by direct inspection of the payload construction. apply_job_fail_transition() (job.c, the guarded core BOTH the live job_ctrl path (mark_job_fail) and the deferred apply seam (apply_committed_entry's WAL_JOB_FAIL case) invoke) unconditionally executes `job->job.finish_time = time(NULL);` with no external value ever consulted, at APPLY time -- meaning even a caller willing to plumb a real value through record_job_fail_fact's WAL payload would still need a apply_job_fail_transition() signature change to have it actually land in job.job.finish_time. This is a pre-existing, already-documented residual: job.c's own comment near apply_job_postdone_transition (task 4.1) already states 'finish_time is not part of the wal_jobstate_t wire format ... has no apply-time equivalent' for a sibling fact type, confirming this is systemic to the wal_jobstate_t-based fact family, not something worker-rejoin-resync introduced. Fixing it for real would require extending WAL_JOB_FAIL's wire payload (wal_types.h, mirroring bug-1689's own WAL_JOB_FINISH trailing-fields precedent) PLUS an apply_job_fail_transition() signature/behavior change reaching every fail-reporting call site in the system (job_ctrl's mark_job_fail, on_job_fail's ordinary worker-fail receipt path, and this feature's own reconciliation caller) -- real, broad-blast-radius work, not a narrow patch.

## Fix
SPLIT (2026-09-01, coordinator-directed): the FINISH-success half of this gap is FIXED as part of worker-rejoin-resync task 3.2 -- added record_job_finish_fact_at(cluster_t*, job_t*, xce_jobfinish_t*, uint64_t finish_time_ms) (job.c), record_job_finish_fact()'s own worker-reported-finish-time sibling (identical guard/hook-fire/durable-append body, differing only in passing the caller-supplied finish_time -- converted ms->s -- to persist_submit_job_finish_fact() instead of time(NULL)); record_job_finish_fact() itself is UNMODIFIED, its own caller (on_job_finish, an ordinary near-instantaneous worker report) correctly keeps using receipt-time. resync_apply_report()'s finishedOk branch now calls record_job_finish_fact_at(c, job, &finish, entry->finish_time_ms). Verified via a real restart-replay test (test_apply_report_finished_success_uses_worker_reported_finish_time, test_worker_resync.c): a finished-success entry with finish_time_ms set to 'now minus 1 day' recovers job.finish_time equal to that worker-reported value after persist_recover(), not reconciliation-processing wall-clock time. This was possible narrowly for FINISH because persist_submit_job_finish_fact() already accepted an explicit finish_time parameter at the primitive level -- record_job_finish_fact()'s specific call site just never exposed it. The FAIL half (see root_cause) is NOT YET FIXED and is NOT task 3.2's (or any single narrow task's) job to fix -- it needs the broader WAL_JOB_FAIL payload extension + apply_job_fail_transition() signature change described above, deliberately deferred pending a dedicated task, matching the same 'deliberately deferred, documented, not blocking' treatment bug-1631's own sibling gap received earlier in this same feature. The reconciled job's STATE (EXIT) is always correct for a finished-failure outcome either way -- only its recorded finish_time reflects apply-processing time rather than the worker's true report.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1689, bug-1678, bug-1746, bug-1631