lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): WAL_JOB_FINISH loses exit_info/finish_time on replay -- and exit_info was never set at all in production (bug-1689)

Browse source 
Same class as bug-1678/bug-1687: record_job_finish_fact() wrote only
{jobid, state} for WAL_JOB_FINISH. Both apply seams either fabricated
finish_time via time(NULL) or touched nothing beyond state.

Worse than "lost on restart": apply_job_finish_transition() -- the only
function that ever set job.exit_info -- is dead code on the real
production apply path. The raft-apply-decoupling redesign routes actual
mutation through the leaner apply_job_update_transition() instead, so
exit_info was never populated at all, restart or not, before this fix.

job_exitinfo_t ({int exitcode;}) has no existing capnp mapping, so this
uses a small native (non-capnp) trailing payload -- wal_jobstate_t header
+ uint64_t finish_time + int32_t exit_code -- mirroring wal_jobstate_t's
own native-struct style rather than inventing a capnp wrapper for one
field. run_time is never carried on the wire, always re-derived as
finish_time - start_time at apply time. Both apply seams updated;
old-format payloads (exactly sizeof(wal_jobstate_t)) decode unchanged
(fabricated finish_time, NULL exit_info) -- full backward compatibility.

Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension). 6 new tests in test_wal_job_finish_fact.c; RED/GREEN
independently verified (4 genuine pre-fix failures with real
expected-vs-fabricated mismatches, 2 backward-compat tests correctly
passed both before and after). Full controller suite: 1152 tests, 2
failures -- both the pre-existing, unrelated bug-1561 overlayfs sandbox
flake (confirmed flaky across repeated re-runs of the same binary).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 09:58:55 +00:00
parent 44bc50202a
commit c2690cbf95
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
8 changed files with 1381 additions and 425 deletions
Download patch file Download diff file Expand all files Collapse all files

65
src/controller/job.c
View file

@ -851,6 +851,45 @@ int restore_job_start_state(cluster_t* c, uint64_t jobid, uint32_t state,
return 0;
}
/* bug-1689: WAL_JOB_FINISH's own sibling of restore_job_start_state()/
* restore_job_dispatch_state() above (bug-1687/bug-1678), for
* apply_wal_entry()'s standby catch-up / replicated-apply seam -- takes
* ownership of an optional decoded job_exitinfo_t (NULL for a state-only
* WAL_JOB_FINISH entry, e.g. one written before this fix) and a
* wire-carried finish_time (0 meaning "not carried -- leave
* job.finish_time/run_time untouched", matching this seam's own pre-fix
* behavior of never touching them for WAL_JOB_FINISH at all) so a promoted
* standby recovers the worker-confirmed exit_code and the true finish_time
* the same way apply_committed_entry's WAL_JOB_FINISH case already does for
* the primary's own boot replay. job_exitinfo_t has no nested pointers (a
* single int), so it is released with plain FREEUP, not a capnp
* free_*_ptr helper. */
int restore_job_finish_state(cluster_t* c, uint64_t jobid, uint32_t state,
uint64_t finish_time, job_exitinfo_t* exit_info)
{
job_t* j = NULL;
lock(c->jobmgr, RTTI_LOCK_READ);
key_find(c->jobmgr, (void*)jobid, (void**)&j);
unlock(c->jobmgr);
if (!j) {
FREEUP(exit_info);
return -ENOENT;
}
lock(j, RTTI_LOCK_WRITE);
j->job.state = (int)state;
if (finish_time != 0)
j->job.finish_time = (time_t)finish_time;
if (exit_info != NULL) {
FREEUP(j->job.exit_info);
j->job.exit_info = exit_info;
}
if (finish_time != 0)
j->job.run_time = (int)(j->job.finish_time - j->job.start_time);
unlock(j);
return 0;
}
static int get_nslots_in_alloc(xce_host_alloc_t* alloc)
{
int i;
@ -2349,10 +2388,28 @@ static int record_job_finish_fact(cluster_t* c, job_t* job,
/* Durable append only — no mutation, no AFTER_EVENT firing here (Req
* 5.1, 5.3). The resultant state is deterministic from finish->exit_code
* alone, independent of current job state. */
(void)persist_submit_job_fact(&c->persist, WAL_JOB_FINISH, job->job.jobid,
(finish->exit_code == 0) ? JOB_STAT_DONE
: JOB_STAT_EXIT);
* alone, independent of current job state.
*
* bug-1689: persist_submit_job_finish_fact() carries a receipt-time
* finish_time and the worker-reported exit_code alongside {jobid,
* state} -- see that function's doc comment and wal_types.h's
* WAL_JOB_FINISH payload note for the full rationale (mirrors
* bug-1687's WAL_JOB_START fix). Before this fix, persist_submit_job_
* fact()'s bare {jobid, state} payload silently dropped both: job.
* exit_info replayed permanently NULL and job.finish_time (and its
* derived job.run_time) were re-fabricated (time(NULL)) fresh at every
* apply instead of carrying the true receipt-time value. Capturing
* finish_time HERE, once, at receipt time (rather than re-deriving
* time(NULL) independently at each of apply_committed_entry's/
* apply_wal_entry's OWN apply-time moments) makes it the single
* authoritative value both apply seams converge on. */
time_t finish_time = time(NULL);
(void)persist_submit_job_finish_fact(&c->persist, job->job.jobid,
(finish->exit_code == 0)
? JOB_STAT_DONE
: JOB_STAT_EXIT,
(uint64_t)finish_time,
finish->exit_code);
return 0;
}