lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): WAL_JOB_FINISH loses exit_info/finish_time on replay -- and exit_info was never set at all in production (bug-1689)

Browse source 
Same class as bug-1678/bug-1687: record_job_finish_fact() wrote only
{jobid, state} for WAL_JOB_FINISH. Both apply seams either fabricated
finish_time via time(NULL) or touched nothing beyond state.

Worse than "lost on restart": apply_job_finish_transition() -- the only
function that ever set job.exit_info -- is dead code on the real
production apply path. The raft-apply-decoupling redesign routes actual
mutation through the leaner apply_job_update_transition() instead, so
exit_info was never populated at all, restart or not, before this fix.

job_exitinfo_t ({int exitcode;}) has no existing capnp mapping, so this
uses a small native (non-capnp) trailing payload -- wal_jobstate_t header
+ uint64_t finish_time + int32_t exit_code -- mirroring wal_jobstate_t's
own native-struct style rather than inventing a capnp wrapper for one
field. run_time is never carried on the wire, always re-derived as
finish_time - start_time at apply time. Both apply seams updated;
old-format payloads (exactly sizeof(wal_jobstate_t)) decode unchanged
(fabricated finish_time, NULL exit_info) -- full backward compatibility.

Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension). 6 new tests in test_wal_job_finish_fact.c; RED/GREEN
independently verified (4 genuine pre-fix failures with real
expected-vs-fabricated mismatches, 2 backward-compat tests correctly
passed both before and after). Full controller suite: 1152 tests, 2
failures -- both the pre-existing, unrelated bug-1561 overlayfs sandbox
flake (confirmed flaky across repeated re-runs of the same binary).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 09:58:55 +00:00
parent 44bc50202a
commit c2690cbf95
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
8 changed files with 1381 additions and 425 deletions
Download patch file Download diff file Expand all files Collapse all files

20
src/include/persist.h
View file

@ -1437,6 +1437,26 @@ int persist_submit_job_start_fact(persist_t* p, uint64_t jobid, int state,
uint64_t start_time,
const job_pidinfo_t* info);
/* bug-1689: WAL_JOB_FINISH's own sibling of persist_submit_job_fact(),
* mirroring persist_submit_job_start_fact() (bug-1687) exactly but for the
* worker-confirmed-finish fact -- additionally encoding a receipt-time
* finish_time AND the worker-reported exit_code into the fact's payload
* (header {jobid, state} + a fixed uint64_t finish_time + a fixed int32_t
* exit_code) so a restart/catch-up replay can reconstruct job.exit_info and
* the true job.finish_time instead of leaving exit_info permanently NULL and
* finish_time fabricated fresh (time(NULL)) at every apply. job_exitinfo_t
* is a single int with no existing Cap'n Proto mapping (no JobExitInfo
* struct in controller.capnp) -- this is a small NATIVE trailing payload,
* mirroring wal_jobstate_t's own native-struct style, not a capnp wrapper
* (unlike persist_submit_job_start_fact()'s variable-length JobPIDInfo,
* which genuinely needed one). run_time is NOT carried on the wire -- it is
* a pure derivation (finish_time - start_time), re-derived at each apply
* seam once both are known, mirroring apply_job_finish_transition()'s own
* formula. Returns the local-append rc, same contract as
* persist_submit_job_fact(). */
int persist_submit_job_finish_fact(persist_t* p, uint64_t jobid, int state,
uint64_t finish_time, int exit_code);
/* bug-1679: fire XCE_CONTROLLER_HOOK_JOB's HOOK_DETAIL_JOB_UPDATE AFTER_EVENT
* for a job whose durable WAL fact the caller has ALREADY appended itself
* (e.g. dispatch_a_job_()'s own persist_submit_job_dispatch_fact() call,

19
src/include/wal_types.h
View file

@ -112,7 +112,24 @@ typedef enum {
* seams treat a payload of exactly
* sizeof(wal_jobstate_t) as the legacy state-only
* shape for full backward compatibility. */
WAL_JOB_FINISH = 3,
WAL_JOB_FINISH = 3, /* worker-confirmed job finish. Payload is
* wal_jobstate_t{jobid, state=DONE|EXIT, _pad=0}
* for any entry written before bug-1689; as of
* bug-1689 the real producer
* (persist_submit_job_finish_fact) appends a fixed
* uint64_t finish_time (receipt-time clock, not
* fabricated at apply time) plus a fixed int32_t
* exit_code -- both apply seams treat a payload of
* exactly sizeof(wal_jobstate_t) as the legacy
* state-only shape for full backward
* compatibility. job_exitinfo_t is a single int
* with no existing Cap'n Proto mapping, so this is
* a small NATIVE trailing payload (mirroring
* wal_jobstate_t's own native-struct style), not a
* capnp wrapper. run_time is NOT carried on the
* wire -- it is a pure derivation (finish_time -
* start_time), re-derived at each apply seam once
* both are known. */
WAL_JOB_FAIL = 4,
WAL_JOB_STOP = 5,
WAL_JOB_RESUME = 6,