lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): WAL_JOB_START loses pidinfo/start_time on restart replay (bug-1687)

Browse source 
Same class as bug-1678: apply_job_start_transition() sets the real
job->job.pidinfo (worker-reported PIDs/PGIDs/exdata) and start_time, but
the WAL fact for WAL_JOB_START only carried the shared wal_jobstate_t
{jobid, state} header -- both apply seams (apply_committed_entry() for
primary boot replay, apply_wal_entry() for standby catch-up) restored
state only. pidinfo stayed permanently NULL after a restart; start_time
was fabricated with time(NULL) at replay instead of being carried,
drifting from the true value.

Mirrors bug-1678's exact fix shape: capnp_encode_pidinfo()/decode_pidinfo()
(persist.c, using the pre-existing JobPIDInfo capnp struct -- no schema
change needed) serialize job_pidinfo_t into the WAL payload, trailing a
new uint64_t start_time field appended after wal_jobstate_t.
persist_submit_job_start_fact() replaces the old state-only
persist_submit_job_fact() call for this type. Both apply seams updated;
a payload of exactly sizeof(wal_jobstate_t) (any entry written before
this fix) decodes with pidinfo left NULL and start_time fabricated via
time(NULL), exactly matching pre-fix behavior -- full backward
compatibility, verified by every pre-existing WAL_JOB_START test using
the old bare-struct shape.

Also fixed a genuine, unrelated deadlock this change's larger payload
exposed in test_commit_barrier_reply.c: cbrp_count_jobstate_entries_ex()
matched WAL entries by exact payload size (==), so once WAL_JOB_START
legitimately grew it silently undercounted to 0 and a downstream test
spun forever polling an uncounted target. Fixed the match to >= (jobid/
state always live at a fixed leading offset) -- bisected via /proc/*/wchan
futex inspection (no ptrace in this sandbox), confirmed as the actual
cause, not the teardown-ordering red herring first suspected.

Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension, added after bug-1678). 6 new tests in
test_wal_job_start_fact.c; RED/GREEN independently verified (4 tests
genuinely failed pre-fix with exact expected-vs-fabricated value
mismatches, 2 backward-compat tests correctly passed both before and
after). Full controller suite: 1146 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 09:23:00 +00:00
parent bc3f76b42e
commit 955efecd19
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
9 changed files with 992 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

16
src/include/persist.h
View file

@ -1421,6 +1421,22 @@ int persist_submit_job_fact_ex(persist_t* p, uint32_t wal_type,
int persist_submit_job_dispatch_fact(persist_t* p, uint64_t jobid, int state,
const xce_job_alloc_t* alloc);
/* bug-1687: WAL_JOB_START's own sibling of persist_submit_job_fact(),
* mirroring persist_submit_job_dispatch_fact() (bug-1678) exactly but for
* the worker-confirmed-start fact -- additionally encoding the
* worker-reported job_pidinfo_t (real pids/pgids/exdata) AND a receipt-time
* start_time into the fact's payload (header {jobid, state} + a fixed
* uint64_t start_time + trailing Cap'n Proto JobPIDInfo bytes) so a
* restart/catch-up replay can reconstruct job.pidinfo and the true
* job.start_time instead of leaving pidinfo permanently NULL and
* start_time fabricated fresh (time(NULL)) at every replay. `info` must be
* non-NULL (the real call site, record_job_start_fact(), always has one by
* construction -- it is the worker's START report itself). Returns the
* local-append rc, same contract as persist_submit_job_fact(). */
int persist_submit_job_start_fact(persist_t* p, uint64_t jobid, int state,
uint64_t start_time,
const job_pidinfo_t* info);
/* bug-1679: fire XCE_CONTROLLER_HOOK_JOB's HOOK_DETAIL_JOB_UPDATE AFTER_EVENT
* for a job whose durable WAL fact the caller has ALREADY appended itself
* (e.g. dispatch_a_job_()'s own persist_submit_job_dispatch_fact() call,

12
src/include/wal_types.h
View file

@ -101,7 +101,17 @@ _Static_assert(sizeof(wal_entry_hdr_t) == 40,
* ============================================================ */
typedef enum {
WAL_JOB_SUBMIT = 1, /* SYNC replication */
WAL_JOB_START = 2,
WAL_JOB_START = 2, /* worker-confirmed job start. Payload is
* wal_jobstate_t{jobid, state=RUN, _pad=0} for any
* entry written before bug-1687; as of bug-1687 the
* real producer (persist_submit_job_start_fact)
* appends a fixed uint64_t start_time (receipt-time
* clock, not fabricated at apply time) plus
* trailing Cap'n Proto JobPIDInfo bytes (the
* worker-reported pids/pgids/exdata) -- both apply
* seams treat a payload of exactly
* sizeof(wal_jobstate_t) as the legacy state-only
* shape for full backward compatibility. */
WAL_JOB_FINISH = 3,
WAL_JOB_FAIL = 4,
WAL_JOB_STOP = 5,