lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): job.start_time never set -- dispatch/WAL_JOB_START RUN-state collision (bug-1697)

Browse source 
dispatch_a_job_() synchronously sets state=JOB_STAT_RUN at dispatch time
(scheduler-dispatch-commit task 3.1's deliberate no-worker-ack-wait
semantic). By the time the worker's later WAL_JOB_START fact commits,
apply_job_update_transition() sees state already == RUN and returns
NOOP_DUPLICATE instead of APPLIED -- so bug-1687's start_time/pidinfo
decode/adopt block, gated on result==APPLIED, could never run for any
job. start_time (and consequently run_time) stayed 0 forever. Same bug
class already fixed for WAL_JOB_RUSAGE/CKPT/PREASON in this identical
apply function; WAL_JOB_START was missed because bug-1687 predated the
dispatch-time RUN semantic change.

Moves start_time to dispatch-owned: dispatch_a_job_() now stamps
job.start_time alongside state=RUN, under the same write lock.
WAL_JOB_DISPATCH gains a flag-gated (WAL_JOB_DISPATCH_HAS_START_TIME_FLAG)
start_time field so it survives replay/restart; both apply seams
(apply_committed_entry, apply_wal_entry) and restore_job_dispatch_state
decode it when present, falling through to the old alloc-only shape
otherwise. WAL_JOB_START's own now-permanently-dead start_time write is
removed; pidinfo adoption made unconditional to match.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-27 08:25:40 +00:00
parent faedc6fc49
commit ac1e312e2f
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
9 changed files with 859 additions and 184 deletions
Download patch file Download diff file Expand all files Collapse all files

53
src/controller/job.c
View file

@ -790,9 +790,16 @@ int restore_job_state(cluster_t* c, uint64_t jobid, uint32_t state)
* ownership of an optional decoded resource allocation (NULL for a
* state-only WAL_JOB_DISPATCH entry, e.g. one written before this fix) so a
* promoted standby recovers EXEC_HOST the same way apply_committed_entry's
* WAL_JOB_DISPATCH case already does for the primary's own boot replay. */
* WAL_JOB_DISPATCH case already does for the primary's own boot replay.
*
* bug-1697: also takes a wire-carried `start_time` (0 meaning "not carried
* -- leave job.start_time untouched", matching every sibling restore_job_*
* function's own "0 == not carried" convention). job.start_time is now
* DISPATCH-owned (see wal_types.h's WAL_JOB_START doc comment for the full
* rationale), so this is the seam that must restore it on a promoted
* standby, not WAL_JOB_START's own restore_job_start_state() below. */
int restore_job_dispatch_state(cluster_t* c, uint64_t jobid, uint32_t state,
xce_job_alloc_t* alloc)
uint64_t start_time, xce_job_alloc_t* alloc)
{
job_t* j = NULL;
lock(c->jobmgr, RTTI_LOCK_READ);
@ -806,6 +813,8 @@ int restore_job_dispatch_state(cluster_t* c, uint64_t jobid, uint32_t state,
lock(j, RTTI_LOCK_WRITE);
j->job.state = (int)state;
if (start_time != 0)
j->job.start_time = (time_t)start_time;
if (alloc != NULL) {
if (j->job.allocs != NULL)
free_JobAlloc_ptr(&(j->job.allocs));
@ -819,14 +828,17 @@ int restore_job_dispatch_state(cluster_t* c, uint64_t jobid, uint32_t state,
* above (bug-1678), for apply_wal_entry()'s standby catch-up / replicated-
* apply seam -- takes ownership of an optional decoded job_pidinfo_t (NULL
* for a state-only WAL_JOB_START entry, e.g. one written before this fix)
* and a wire-carried start_time (0 meaning "not carried -- leave
* job.start_time untouched", matching this seam's own pre-fix behavior of
* never stamping it for WAL_JOB_START at all) so a promoted standby
* recovers the worker-confirmed pids/pgids/exdata and the true start_time
* the same way apply_committed_entry's WAL_JOB_START case already does for
* the primary's own boot replay. */
* so a promoted standby recovers the worker-confirmed pids/pgids/exdata the
* same way apply_committed_entry's WAL_JOB_START case already does for the
* primary's own boot replay.
*
* bug-1697: no longer takes/touches start_time at all -- job.start_time is
* now exclusively DISPATCH-owned (restore_job_dispatch_state() above), so
* this function must leave it alone entirely rather than risk clobbering
* the correct dispatch-time value with a stale worker-ack-time one (see
* wal_types.h's WAL_JOB_START doc comment for the full rationale). */
int restore_job_start_state(cluster_t* c, uint64_t jobid, uint32_t state,
uint64_t start_time, job_pidinfo_t* pidinfo)
job_pidinfo_t* pidinfo)
{
job_t* j = NULL;
lock(c->jobmgr, RTTI_LOCK_READ);
@ -840,8 +852,6 @@ int restore_job_start_state(cluster_t* c, uint64_t jobid, uint32_t state,
lock(j, RTTI_LOCK_WRITE);
j->job.state = (int)state;
if (start_time != 0)
j->job.start_time = (time_t)start_time;
if (pidinfo != NULL) {
if (j->job.pidinfo != NULL)
free_JobPIDInfo_ptr(&(j->job.pidinfo));
@ -1685,6 +1695,18 @@ static int dispatch_a_job_(cluster_t* c, endpoint_t* ep, xce_job_alloc_t* alloc)
j->job.state = JOB_STAT_RUN;
/* bug-1697: start_time is now stamped HERE, synchronously alongside
* the RUN transition, rather than left to the worker's later
* WAL_JOB_START report. dispatch_a_job_() setting job.state =
* JOB_STAT_RUN synchronously means any WAL_JOB_START fact for this
* job always finds state already == RUN by the time it applies
* (apply_job_update_transition() reports JOB_TRANSITION_NOOP_
* DUPLICATE), so it can never be the genuine first RUN transition
* and start_time adoption gated on that transition would be
* permanently dead code -- see wal_types.h's WAL_JOB_START doc
* comment for the full rationale. */
j->job.start_time = time(NULL);
/* Reset the consecutive-retryable-failure counter (task 1.3): a
* fresh successful dispatch clears any prior failure history, so a
* job that later fails again is evaluated against the bound as if
@ -1700,9 +1722,14 @@ static int dispatch_a_job_(cluster_t* c, endpoint_t* ep, xce_job_alloc_t* alloc)
* shutdown. j->job.allocs is read here, still under this same
* write lock, as the freshly-constructed canonical struct just
* stored above -- not the (now partially moved-from) local
* `alloc`/`allocs` variables. */
* `alloc`/`allocs` variables. bug-1697: also carries j->job.
* start_time, the same snapshot just stamped above under this same
* write lock, so a restart replay recovers the true dispatch-time
* clock instead of leaving it at 0 forever. */
(void)persist_submit_job_dispatch_fact(&c->persist, alloc->jobid,
JOB_STAT_RUN, j->job.allocs);
JOB_STAT_RUN,
(uint64_t)j->job.start_time,
j->job.allocs);
unlock(j);