---
id: "bug-1697"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-08-25T20:22:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/job.c", "src/controller/persist.c"]
tags: ["controller", "job-lifecycle", "wal-dispatch", "wal-job-start", "start-time", "pidinfo", "field-parity", "apply-committed-entry", "transition-noop-collision", "found-live-no-restart", "bug-1679-adjacent", "bug-1687-adjacent", "bug-1678-adjacent", "critical"]
related: ["bug-1679", "bug-1687", "bug-1678", "bug-1690", "bug-1691", "bug-1692"]
legacy: true
---

# bug-1697: Job start_time is always 0 for every job, live, no restart needed (observed via xce_test REPL…

## Description
Job start_time is always 0 for every job, live, no restart needed (observed via xce_test REPL joblist during an HA soak). run_time (finish_time - start_time) is consequently also wrong for every finished job.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
dispatch_a_job_() (job.c:1686, scheduler-dispatch-commit task 3.1) synchronously sets j->job.state = JOB_STAT_RUN at DISPATCH time, before the worker ever reports back -- this is a deliberate, already-made semantic change: the controller decides RUN at dispatch, it no longer waits for a worker ack. When the worker's LATER on_job_start()/WAL_JOB_START fact commits and apply_committed_entry()'s live apply pump (persist.c, shared WAL_JOB_START/FINISH/STOP/RESUME/RUSAGE/CKPT/PREASON/DISPATCH case group starting ~persist.c:4430) processes it, apply_job_update_transition(j, JOB_STAT_RUN) (job.c:2779) sees job->job.state already == JOB_STAT_RUN (set by dispatch, moments/seconds earlier) and returns JOB_TRANSITION_NOOP_DUPLICATE (job.c:2795-2799), never JOB_TRANSITION_APPLIED. The wire-carried start_time/pidinfo decode/adopt block bug-1687 added (persist.c ~4602-4637) is gated on `result == JOB_TRANSITION_APPLIED` (persist.c:4594), so for WAL_JOB_START it can now NEVER run, for any job -- start_time and pidinfo (pids/pgids/exdata) both silently stay at their zero-initialized defaults forever. This is the exact same class of bug already fixed for WAL_JOB_RUSAGE/WAL_JOB_CKPT/WAL_JOB_PREASON in this identical function (their own comments describe it verbatim: 'gating this adoption on result==JOB_TRANSITION_APPLIED ... would make it dead code ... decode/adopt unconditionally') -- WAL_JOB_START was never given the same treatment because bug-1687 (which added the start_time/pidinfo wire fields) predates/was unaware of the dispatch-time RUN semantic collision. bug-1679's own comment (job.c:1709-1719) already independently documented and worked around the SAME root collision for a different symptom (the apiserver update-notification hook silently not firing on this same NOOP_DUPLICATE path) -- the two symptoms were never connected as the same root cause until now.

## Fix
FIXED -- start_time moved to dispatch-owned, pidinfo adoption made unconditional, exactly the confirmed scope. (1) job.c dispatch_a_job_() (job.c:1696-1708) stamps j->job.start_time = time(NULL) synchronously alongside j->job.state = JOB_STAT_RUN, under the same write lock, before j->job.allocs/persist_submit_job_dispatch_fact(). (2) wal_types.h adds WAL_JOB_DISPATCH_HAS_START_TIME_FLAG (new _pad bit, mirroring WAL_JOB_FAIL_STAGE_FLAG/WAL_JOB_UPDATE_POSTDONE_MARK). persist_submit_job_dispatch_fact() (persist.c:7670, persist.h:1465) gained a `uint64_t start_time` parameter; payload is now [wal_jobstate_t header, _pad=FLAG][uint64_t start_time][JobAlloc bytes] -- flag-gated, not length-based, since JobAlloc's own variable length would make a length heuristic ambiguous against the old header+alloc-only shape. (3) Both apply seams' WAL_JOB_DISPATCH cases (apply_committed_entry persist.c:4716 branch; apply_wal_entry persist.c:4183 case; restore_job_dispatch_state job.c:801) decode/adopt start_time when the flag is set, else fall through to the pre-fix alloc-only decode untouched. (4) apply_committed_entry's WAL_JOB_START handling (persist.c:4627-4660): start_time touching removed entirely (dead-code branch and its time(NULL) fallback deleted); pidinfo decode/adopt moved OUT of the `result == JOB_TRANSITION_APPLIED` gate to run unconditionally, mirroring the RUSAGE/CKPT/PREASON blocks in the same function. apply_wal_entry's WAL_JOB_START case (persist.c:4105) and restore_job_start_state() (job.c:840, dropped the start_time parameter) likewise stop touching start_time; pidinfo there was already unconditional (this seam never calls apply_job_update_transition, so it never had the NOOP-gating hazard). persist_submit_job_start_fact()'s wire format is UNCHANGED (still writes start_time for on-disk layout continuity) -- only the READ side stopped consuming it. Pre-fix WAL_JOB_DISPATCH/WAL_JOB_START entries (_pad flag unset) fall back to the exact pre-fix behavior, an accepted one-time migration-boundary gap consistent with every other fix in this campaign. TDD: RED confirmed by temporarily reverting just the two defect lines (job.c's start_time stamp; persist.c's pidinfo gate) and running the full suite -- 5 genuine new failures (test_dc_success_stamps_start_time_synchronously, test_dc_success_appends_wal_job_dispatch_durably, test_wjs_apply_committed_entry_adopts_pidinfo_when_already_run_from_dispatch, test_wjs_restart_replay_recovers_dispatch_start_time_and_started_pidinfo, test_ace_job_start_sets_start_time) plus the pre-existing baseline (2 known bug-1561 test_config_repl.c failures + 1 concurrent bug-1696 test_job_array_replay_fence.c failure, both left untouched). GREEN after restoring the fix: rebuilt clean (0 compile errors) in an isolated dir (avoided the live HA soak's build-asan), full controller_unit_test run 3x direct (ASAN_OPTIONS=detect_leaks=0, matching this binary's own meson.build env) -- consistently "1198 Tests 3 Failures 0 Ignored", exactly the known baseline, 0 new failures. Live manual verification: single-node controller+scheduler+xcecd_virtual worker deployed via deploy-ha.sh (isolated port 19991, isolated workdir, no interference with the concurrent bug-1696 soak); 4 real xce_test jobsub->dispatch->finish cycles via the REPL all showed nonzero StartTime in `joblist` immediately (e.g. jobid 0-14-...-55018: StartTime 1787718938, matching SubmitTime). pidinfo's live collision path (jstart is a worker-mode-only REPL command, racy/awkward to trigger against a job a real worker is also concurrently finishing) was left to the deterministic unit tests above rather than a flaky live race -- covered exhaustively there at the exact apply_committed_entry()/on_job_start() call sites. Files: src/controller/job.c, src/controller/persist.c, src/include/persist.h, src/include/wal_types.h, src/controller/testsuite/{test_wal_job_dispatch_fact.c,test_wal_job_start_fact.c,test_dispatch_commit.c,test_apply_committed_entry.c,test_repl_apply.c (weak stub signature only)}.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1679, bug-1687, bug-1678, bug-1690, bug-1691, bug-1692