lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): WAL_JOB_START loses pidinfo/start_time on restart replay (bug-1687)

Browse source 
Same class as bug-1678: apply_job_start_transition() sets the real
job->job.pidinfo (worker-reported PIDs/PGIDs/exdata) and start_time, but
the WAL fact for WAL_JOB_START only carried the shared wal_jobstate_t
{jobid, state} header -- both apply seams (apply_committed_entry() for
primary boot replay, apply_wal_entry() for standby catch-up) restored
state only. pidinfo stayed permanently NULL after a restart; start_time
was fabricated with time(NULL) at replay instead of being carried,
drifting from the true value.

Mirrors bug-1678's exact fix shape: capnp_encode_pidinfo()/decode_pidinfo()
(persist.c, using the pre-existing JobPIDInfo capnp struct -- no schema
change needed) serialize job_pidinfo_t into the WAL payload, trailing a
new uint64_t start_time field appended after wal_jobstate_t.
persist_submit_job_start_fact() replaces the old state-only
persist_submit_job_fact() call for this type. Both apply seams updated;
a payload of exactly sizeof(wal_jobstate_t) (any entry written before
this fix) decodes with pidinfo left NULL and start_time fabricated via
time(NULL), exactly matching pre-fix behavior -- full backward
compatibility, verified by every pre-existing WAL_JOB_START test using
the old bare-struct shape.

Also fixed a genuine, unrelated deadlock this change's larger payload
exposed in test_commit_barrier_reply.c: cbrp_count_jobstate_entries_ex()
matched WAL entries by exact payload size (==), so once WAL_JOB_START
legitimately grew it silently undercounted to 0 and a downstream test
spun forever polling an uncounted target. Fixed the match to >= (jobid/
state always live at a fixed leading offset) -- bisected via /proc/*/wchan
futex inspection (no ptrace in this sandbox), confirmed as the actual
cause, not the teardown-ordering red herring first suspected.

Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension, added after bug-1678). 6 new tests in
test_wal_job_start_fact.c; RED/GREEN independently verified (4 tests
genuinely failed pre-fix with exact expected-vs-fabricated value
mismatches, 2 backward-compat tests correctly passed both before and
after). Full controller suite: 1146 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 09:23:00 +00:00
parent bc3f76b42e
commit 955efecd19
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
9 changed files with 992 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

58
src/controller/job.c
View file

@ -815,6 +815,42 @@ int restore_job_dispatch_state(cluster_t* c, uint64_t jobid, uint32_t state,
return 0;
}
/* bug-1687: WAL_JOB_START's own sibling of restore_job_dispatch_state()
* above (bug-1678), for apply_wal_entry()'s standby catch-up / replicated-
* apply seam -- takes ownership of an optional decoded job_pidinfo_t (NULL
* for a state-only WAL_JOB_START entry, e.g. one written before this fix)
* and a wire-carried start_time (0 meaning "not carried -- leave
* job.start_time untouched", matching this seam's own pre-fix behavior of
* never stamping it for WAL_JOB_START at all) so a promoted standby
* recovers the worker-confirmed pids/pgids/exdata and the true start_time
* the same way apply_committed_entry's WAL_JOB_START case already does for
* the primary's own boot replay. */
int restore_job_start_state(cluster_t* c, uint64_t jobid, uint32_t state,
uint64_t start_time, job_pidinfo_t* pidinfo)
{
job_t* j = NULL;
lock(c->jobmgr, RTTI_LOCK_READ);
key_find(c->jobmgr, (void*)jobid, (void**)&j);
unlock(c->jobmgr);
if (!j) {
if (pidinfo != NULL)
free_JobPIDInfo_ptr(&pidinfo);
return -ENOENT;
}
lock(j, RTTI_LOCK_WRITE);
j->job.state = (int)state;
if (start_time != 0)
j->job.start_time = (time_t)start_time;
if (pidinfo != NULL) {
if (j->job.pidinfo != NULL)
free_JobPIDInfo_ptr(&(j->job.pidinfo));
j->job.pidinfo = pidinfo;
}
unlock(j);
return 0;
}
static int get_nslots_in_alloc(xce_host_alloc_t* alloc)
{
int i;
@ -2190,9 +2226,25 @@ static int record_job_start_fact(cluster_t* c, job_t* job, job_pidinfo_t* info)
* 5.1, 5.3). JOB_STAT_RUN is the resultant state a START always
* produces once past the guard above; apply_committed_entry() will
* re-validate the same guard against whatever state is current AT
* APPLY TIME before setting it. */
(void)persist_submit_job_fact(&c->persist, WAL_JOB_START, job->job.jobid,
JOB_STAT_RUN);
* APPLY TIME before setting it.
*
* bug-1687: persist_submit_job_start_fact() carries `info` (the real
* worker-reported pids/pgids/exdata) and a receipt-time start_time
* alongside {jobid, state=RUN} -- see that function's doc comment and
* wal_types.h's WAL_JOB_START payload note for the full rationale
* (mirrors bug-1678's WAL_JOB_DISPATCH fix). Before this fix,
* persist_submit_job_fact()'s bare {jobid, state} payload silently
* dropped `info` entirely, so job.pidinfo replayed permanently NULL and
* job.start_time was re-fabricated (time(NULL)) fresh at every replay
* instead of carrying the true worker-reported time. Capturing
* start_time HERE, once, at receipt time (rather than re-deriving
* time(NULL) independently at each of apply_committed_entry's/
* apply_wal_entry's OWN apply-time moments) makes it the single
* authoritative value both apply seams converge on. */
time_t start_time = time(NULL);
(void)persist_submit_job_start_fact(&c->persist, job->job.jobid,
JOB_STAT_RUN, (uint64_t)start_time,
info);
return 0;
}