lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): job.start_time never set -- dispatch/WAL_JOB_START RUN-state collision (bug-1697)

Browse source 
dispatch_a_job_() synchronously sets state=JOB_STAT_RUN at dispatch time
(scheduler-dispatch-commit task 3.1's deliberate no-worker-ack-wait
semantic). By the time the worker's later WAL_JOB_START fact commits,
apply_job_update_transition() sees state already == RUN and returns
NOOP_DUPLICATE instead of APPLIED -- so bug-1687's start_time/pidinfo
decode/adopt block, gated on result==APPLIED, could never run for any
job. start_time (and consequently run_time) stayed 0 forever. Same bug
class already fixed for WAL_JOB_RUSAGE/CKPT/PREASON in this identical
apply function; WAL_JOB_START was missed because bug-1687 predated the
dispatch-time RUN semantic change.

Moves start_time to dispatch-owned: dispatch_a_job_() now stamps
job.start_time alongside state=RUN, under the same write lock.
WAL_JOB_DISPATCH gains a flag-gated (WAL_JOB_DISPATCH_HAS_START_TIME_FLAG)
start_time field so it survives replay/restart; both apply seams
(apply_committed_entry, apply_wal_entry) and restore_job_dispatch_state
decode it when present, falling through to the old alloc-only shape
otherwise. WAL_JOB_START's own now-permanently-dead start_time write is
removed; pidinfo adoption made unconditional to match.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-27 08:25:40 +00:00
parent faedc6fc49
commit ac1e312e2f
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
9 changed files with 859 additions and 184 deletions
Download patch file Download diff file Expand all files Collapse all files

31
src/include/persist.h
View file

@ -1447,8 +1447,23 @@ int persist_submit_job_fact_ex(persist_t* p, uint32_t wal_type,
* itself was never sufficient for this one fact type. `alloc` must be
* non-NULL (the real call site, dispatch_a_job_(), always has one by
* construction). Returns the local-append rc, same contract as
* persist_submit_job_fact(). */
* persist_submit_job_fact().
*
* bug-1697: additionally takes `start_time` (the dispatch-time clock,
* captured once by the caller under the job's write lock -- the same
* snapshot stamped into job.start_time itself, mirroring how
* persist_submit_job_start_fact() below takes its own receipt-time
* start_time). Encoded between the wal_jobstate_t header and the trailing
* JobAlloc bytes, discriminated by the new WAL_JOB_DISPATCH_HAS_START_TIME_
* FLAG bit in wal_jobstate_t._pad (wal_types.h) -- NOT a payload-length
* heuristic, since JobAlloc's own variable length makes old-vs-new length
* ambiguous. job.start_time is now DISPATCH-owned: dispatch_a_job_() sets
* job.state = JOB_STAT_RUN synchronously at DISPATCH time, so any LATER
* WAL_JOB_START fact for the same job always finds state already == RUN by
* the time it applies and can never be the genuine first RUN transition --
* see wal_types.h's WAL_JOB_START doc comment for the full rationale. */
int persist_submit_job_dispatch_fact(persist_t* p, uint64_t jobid, int state,
uint64_t start_time,
const xce_job_alloc_t* alloc);
/* bug-1687: WAL_JOB_START's own sibling of persist_submit_job_fact(),
@ -1457,12 +1472,18 @@ int persist_submit_job_dispatch_fact(persist_t* p, uint64_t jobid, int state,
* worker-reported job_pidinfo_t (real pids/pgids/exdata) AND a receipt-time
* start_time into the fact's payload (header {jobid, state} + a fixed
* uint64_t start_time + trailing Cap'n Proto JobPIDInfo bytes) so a
* restart/catch-up replay can reconstruct job.pidinfo and the true
* job.start_time instead of leaving pidinfo permanently NULL and
* start_time fabricated fresh (time(NULL)) at every replay. `info` must be
* restart/catch-up replay can reconstruct job.pidinfo. `info` must be
* non-NULL (the real call site, record_job_start_fact(), always has one by
* construction -- it is the worker's START report itself). Returns the
* local-append rc, same contract as persist_submit_job_fact(). */
* local-append rc, same contract as persist_submit_job_fact().
*
* bug-1697: `start_time` is still written here (wire layout unchanged, for
* compatibility with any reader of the on-disk format) but is NO LONGER
* adopted by either apply seam -- job.start_time is now exclusively
* DISPATCH-owned (see persist_submit_job_dispatch_fact() above and
* wal_types.h's WAL_JOB_START doc comment for the full rationale). Kept as
* a caller parameter and wire field purely for layout/diagnostic
* continuity, not because it is still consumed. */
int persist_submit_job_start_fact(persist_t* p, uint64_t jobid, int state,
uint64_t start_time,
const job_pidinfo_t* info);

78
src/include/wal_types.h
View file

@ -54,6 +54,32 @@ typedef struct {
* (conditional EXIT backstop + OR JOB_STAT_PDONE) fresh against current
* state. Any other nonzero value is reserved/unused today. */
/* wal_jobstate_t._pad discriminator for WAL_JOB_DISPATCH (bug-1697).
* dispatch_a_job_() (job.c, scheduler-dispatch-commit task 3.1) sets
* job.state = JOB_STAT_RUN synchronously at DISPATCH time, before the
* worker ever reports back — so by the time the LATER WAL_JOB_START fact
* commits and applies, apply_job_update_transition() always finds state
* already == RUN and reports JOB_TRANSITION_NOOP_DUPLICATE, permanently
* disabling any adoption gated on a genuine forward transition for that
* fact type. start_time is therefore moved to be DISPATCH-owned instead:
* WAL_JOB_DISPATCH's payload can now carry a fixed uint64_t start_time
* (the dispatch-time clock, stamped once under the job's write lock)
* between the wal_jobstate_t header and the trailing Cap'n Proto JobAlloc
* bytes (bug-1678). Because JobAlloc's own encoding is variable-length and
* always present in every post-bug-1678 WAL_JOB_DISPATCH entry, a
* length-based "has start_time or not" test would be ambiguous (an
* old-format entry with a large alloc can look identical in length to a
* new-format entry with a small one) — this explicit _pad bit is the
* discriminator instead, mirroring WAL_JOB_FAIL_STAGE_FLAG/WAL_JOB_UPDATE_
* POSTDONE_MARK's own precedent above. _pad is unconditionally 0 for every
* WAL_JOB_DISPATCH entry written before this fix, so introducing this bit
* is safe: a payload with the flag CLEAR decodes exactly the pre-fix shape
* (header immediately followed by JobAlloc bytes, start_time left
* untouched — the accepted one-time migration-boundary gap for any entry
* written by an older binary); a payload with the flag SET decodes
* [header][uint64_t start_time][JobAlloc bytes]. */
#define WAL_JOB_DISPATCH_HAS_START_TIME_FLAG 0x1u
/* WAL_QUEUE_CTRL — queue name is a NUL-terminated string immediately after
* this header. Total payload = sizeof(wal_queuehdr_t) + strlen(name)+1. */
typedef struct {
@ -106,12 +132,26 @@ typedef enum {
* entry written before bug-1687; as of bug-1687 the
* real producer (persist_submit_job_start_fact)
* appends a fixed uint64_t start_time (receipt-time
* clock, not fabricated at apply time) plus
* trailing Cap'n Proto JobPIDInfo bytes (the
* worker-reported pids/pgids/exdata) -- both apply
* seams treat a payload of exactly
* clock) plus trailing Cap'n Proto JobPIDInfo bytes
* (the worker-reported pids/pgids/exdata) -- both
* apply seams treat a payload of exactly
* sizeof(wal_jobstate_t) as the legacy state-only
* shape for full backward compatibility. */
* shape for full backward compatibility.
* bug-1697: the wire-carried start_time field is
* still WRITTEN (wire layout unchanged, so an
* older reader keeps working) but is no longer
* READ/adopted by either apply seam -- dispatch_a_
* job_() setting job.state = JOB_STAT_RUN
* synchronously at DISPATCH time means this fact
* always finds state already == RUN by the time
* it applies, so it can never be the genuine
* first RUN transition; WAL_JOB_DISPATCH (below)
* is now the sole owner/source of job.start_time.
* pidinfo adoption is unaffected and remains
* unconditional (not gated on a forward
* transition), since it is now known this fact's
* transition is a permanent, expected no-op, not
* an edge case. */
WAL_JOB_FINISH = 3, /* worker-confirmed job finish. Payload is
* wal_jobstate_t{jobid, state=DONE|EXIT, _pad=0}
* for any entry written before bug-1689; as of
@ -233,10 +273,30 @@ typedef enum {
* running, distinct from (and independent of)
* WAL_JOB_START's later worker-confirmed-start
* fact. Payload is wal_jobstate_t{jobid, state,
* _pad=0} -- always the legacy absolute-value
* form (no racing same-job fact to defer against,
* since this is the first transition after a job
* leaves PEND). */
* _pad=0} immediately followed by trailing Cap'n
* Proto JobAlloc bytes for any entry written
* before bug-1697 (state is always the legacy
* absolute-value form -- no racing same-job fact
* to defer against, since this is the first
* transition after a job leaves PEND). As of
* bug-1697, dispatch_a_job_() sets job.start_time
* synchronously alongside job.state at DISPATCH
* time (see WAL_JOB_START's own updated comment
* above for why start_time moved here): when
* _pad has WAL_JOB_DISPATCH_HAS_START_TIME_FLAG
* set, the payload is instead wal_jobstate_t
* header + a fixed uint64_t start_time (the
* dispatch-time clock) + the trailing JobAlloc
* bytes. The flag, not payload length, is the
* discriminator -- see
* WAL_JOB_DISPATCH_HAS_START_TIME_FLAG's own doc
* comment above for why a length heuristic would
* be ambiguous against JobAlloc's own variable
* length. Any WAL_JOB_DISPATCH entry written
* before this fix has _pad==0 (flag clear) and
* falls back to the pre-fix state+allocs-only
* behavior, leaving start_time untouched -- an
* accepted one-time migration-boundary gap. */
WAL_QUEUE_CTRL = 20,
WAL_WORKER_CTRL = 30,
WAL_NOOP = 40, /* Internal: empty payload, replay no-op; appended only at promotion */