lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): WAL_JOB_START loses pidinfo/start_time on restart replay (bug-1687)

Browse source 
Same class as bug-1678: apply_job_start_transition() sets the real
job->job.pidinfo (worker-reported PIDs/PGIDs/exdata) and start_time, but
the WAL fact for WAL_JOB_START only carried the shared wal_jobstate_t
{jobid, state} header -- both apply seams (apply_committed_entry() for
primary boot replay, apply_wal_entry() for standby catch-up) restored
state only. pidinfo stayed permanently NULL after a restart; start_time
was fabricated with time(NULL) at replay instead of being carried,
drifting from the true value.

Mirrors bug-1678's exact fix shape: capnp_encode_pidinfo()/decode_pidinfo()
(persist.c, using the pre-existing JobPIDInfo capnp struct -- no schema
change needed) serialize job_pidinfo_t into the WAL payload, trailing a
new uint64_t start_time field appended after wal_jobstate_t.
persist_submit_job_start_fact() replaces the old state-only
persist_submit_job_fact() call for this type. Both apply seams updated;
a payload of exactly sizeof(wal_jobstate_t) (any entry written before
this fix) decodes with pidinfo left NULL and start_time fabricated via
time(NULL), exactly matching pre-fix behavior -- full backward
compatibility, verified by every pre-existing WAL_JOB_START test using
the old bare-struct shape.

Also fixed a genuine, unrelated deadlock this change's larger payload
exposed in test_commit_barrier_reply.c: cbrp_count_jobstate_entries_ex()
matched WAL entries by exact payload size (==), so once WAL_JOB_START
legitimately grew it silently undercounted to 0 and a downstream test
spun forever polling an uncounted target. Fixed the match to >= (jobid/
state always live at a fixed leading offset) -- bisected via /proc/*/wchan
futex inspection (no ptrace in this sandbox), confirmed as the actual
cause, not the teardown-ordering red herring first suspected.

Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension, added after bug-1678). 6 new tests in
test_wal_job_start_fact.c; RED/GREEN independently verified (4 tests
genuinely failed pre-fix with exact expected-vs-fabricated value
mismatches, 2 backward-compat tests correctly passed both before and
after). Full controller suite: 1146 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 09:23:00 +00:00
parent bc3f76b42e
commit 955efecd19
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
9 changed files with 992 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

183
src/controller/persist.c
View file

@ -42,6 +42,16 @@ extern int restore_job_state(cluster_t* c, uint64_t jobid, uint32_t state);
* primary's own boot replay. */
extern int restore_job_dispatch_state(cluster_t* c, uint64_t jobid,
uint32_t state, xce_job_alloc_t* alloc);
/* restore_job_start_state defined in job.c (bug-1687): WAL_JOB_START's own
* sibling of restore_job_state() above, additionally taking ownership of an
* optional decoded job_pidinfo_t and a wire-carried start_time (0 means
* "not carried -- leave job.start_time untouched", the pre-fix payload's
* behavior) so the worker-confirmed start survives standby catch-up /
* replicated-apply the same way apply_committed_entry's WAL_JOB_START case
* already restores it on the primary's own boot replay. */
extern int restore_job_start_state(cluster_t* c, uint64_t jobid,
uint32_t state, uint64_t start_time,
job_pidinfo_t* pidinfo);
extern int worker_from_capnp(cluster_t* c,
const void* capnp_payload, size_t len);
extern int restore_queue_state(cluster_t* c,
@ -198,6 +208,62 @@ static xce_job_alloc_t* decode_joballoc(const uint8_t* payload, size_t len)
return alloc;
}
/* Serialize job_pidinfo_t to a malloc'd Cap'n Proto buffer (bug-1687):
* lets WAL_JOB_START carry the worker-reported pids/pgids/exdata alongside
* the {jobid, state} header, so a restart replay can reconstruct
* job.pidinfo instead of leaving it permanently NULL. Mirrors
* capnp_encode_joballoc above exactly, using the JobPIDInfo struct (already
* in controller.capnp as the existing Job.pidinfo field -- no schema change
* needed here). */
static int capnp_encode_pidinfo(const job_pidinfo_t* info, uint8_t** out,
size_t* out_len)
{
struct capn c;
struct capn_segment* cs;
struct JobPIDInfo cji;
JobPIDInfo_ptr jip;
capn_init_malloc(&c);
cs = capn_root(&c).seg;
encode_JobPIDInfo(cs, &cji, (job_pidinfo_t*)info);
jip = new_JobPIDInfo(cs);
write_JobPIDInfo(&cji, jip);
capn_setp(capn_root(&c), 0, jip.p);
*out_len = capn_size(&c);
*out = malloc(*out_len);
if (!*out) { capn_free(&c); return -ENOMEM; }
capn_write_mem(&c, *out, *out_len, 0);
capn_free(&c);
return 0;
}
/* Decode a Cap'n Proto JobPIDInfo buffer (bug-1687) into a freshly heap-
* allocated job_pidinfo_t*, mirroring decode_joballoc()'s pattern above.
* Returns NULL on a NULL/empty payload; caller owns the result (release via
* free_JobPIDInfo_ptr). */
static job_pidinfo_t* decode_pidinfo(const uint8_t* payload, size_t len)
{
struct capn c_ctx;
JobPIDInfo_ptr jip;
struct JobPIDInfo cji;
job_pidinfo_t* info;
if (!payload || len == 0)
return NULL;
capn_init_mem(&c_ctx, payload, len, 0);
jip.p = capn_getp(capn_root(&c_ctx), 0, 1);
memset(&cji, 0, sizeof(cji));
read_JobPIDInfo(&cji, jip);
info = calloc(1, sizeof(*info));
if (!info) { capn_free(&c_ctx); return NULL; }
decode_JobPIDInfo(info, &cji);
capn_free(&c_ctx);
return info;
}
/* Serialize a host admin-state frame (config-upgrade-snapshot task 2.1, Req
* 1.6/1.7/1.9) to a malloc'd Cap'n Proto buffer. Mirrors capnp_encode_job
* above exactly -- capnp_host_admin_state_t (typemap.h) is a frame-only
@ -3513,7 +3579,6 @@ int apply_wal_entry(persist_t* p, uint32_t type, uint64_t lsn,
restore_job_array(p->cluster, payload, payload_len, lsn, NULL);
break;
case WAL_JOB_START:
case WAL_JOB_FINISH:
case WAL_JOB_FAIL:
case WAL_JOB_STOP:
@ -3529,6 +3594,43 @@ int apply_wal_entry(persist_t* p, uint32_t type, uint64_t lsn,
}
break;
case WAL_JOB_START: {
/* bug-1687: split out of the shared group above -- mirrors
* WAL_JOB_DISPATCH's split just below (bug-1678). This is the one
* fact type in that group whose payload can carry a fixed
* start_time PLUS trailing Cap'n Proto JobPIDInfo bytes
* (persist_submit_job_start_fact's shape) alongside the {jobid,
* state} header, so a promoted standby's catch-up replay recovers
* the worker-confirmed pids/pgids/exdata and the true start_time
* too, not just state. A payload of exactly sizeof(wal_jobstate_t)
* (no trailing bytes -- any WAL_JOB_START entry written before this
* fix) decodes neither, matching the pre-fix state-only behavior
* exactly. */
if (payload_len >= sizeof(wal_jobstate_t)) {
const wal_jobstate_t* js = (const wal_jobstate_t*)payload;
uint64_t start_time = 0;
job_pidinfo_t* pidinfo = NULL;
uint32_t extra = payload_len - (uint32_t)sizeof(wal_jobstate_t);
if (extra >= sizeof(start_time)) {
memcpy(&start_time,
(const uint8_t*)payload + sizeof(wal_jobstate_t),
sizeof(start_time));
uint32_t pidinfo_len = extra - (uint32_t)sizeof(start_time);
if (pidinfo_len > 0) {
pidinfo = decode_pidinfo(
(const uint8_t*)payload + sizeof(wal_jobstate_t)
+ sizeof(start_time),
pidinfo_len);
}
}
restore_job_start_state(p->cluster, js->jobid, js->state,
start_time, pidinfo);
}
break;
}
case WAL_JOB_DISPATCH: {
/* bug-1678: split out of the shared group above -- this is the one
* fact type whose payload can carry a trailing Cap'n Proto JobAlloc
@ -3835,7 +3937,41 @@ int apply_committed_entry(persist_t* p, uint32_t wal_type, uint64_t lsn,
* cores do for their own (currently unreachable on this dispatch
* path) callers. */
if (wal_type == WAL_JOB_START) {
j->job.start_time = time(NULL);
/* bug-1687: a WAL_JOB_START payload can carry a fixed
* start_time plus trailing Cap'n Proto JobPIDInfo bytes
* beyond the wal_jobstate_t header (persist_submit_job_
* start_fact's shape, mirroring bug-1678's WAL_JOB_DISPATCH
* trailing-JobAlloc-bytes handling below). Decode and adopt
* the wire-carried start_time/pidinfo when present; a
* payload of exactly sizeof(wal_jobstate_t) (no trailing
* bytes -- any WAL_JOB_START entry written before this fix)
* intentionally falls through to the old fabricated-
* time(NULL)/pidinfo-untouched behavior for full backward
* compatibility. */
if (len >= sizeof(wal_jobstate_t) + sizeof(uint64_t)) {
uint64_t wire_start_time;
memcpy(&wire_start_time,
(const uint8_t*)payload + sizeof(wal_jobstate_t),
sizeof(wire_start_time));
j->job.start_time = (time_t)wire_start_time;
uint32_t pidinfo_len = len
- (uint32_t)sizeof(wal_jobstate_t)
- (uint32_t)sizeof(uint64_t);
if (pidinfo_len > 0) {
job_pidinfo_t* decoded = decode_pidinfo(
(const uint8_t*)payload + sizeof(wal_jobstate_t)
+ sizeof(uint64_t),
pidinfo_len);
if (decoded != NULL) {
if (j->job.pidinfo != NULL)
free_JobPIDInfo_ptr(&(j->job.pidinfo));
j->job.pidinfo = decoded;
}
}
} else {
j->job.start_time = time(NULL);
}
} else if (wal_type == WAL_JOB_FINISH) {
j->job.finish_time = time(NULL);
j->job.run_time = (int)(j->job.finish_time - j->job.start_time);
@ -6855,6 +6991,49 @@ int persist_submit_job_dispatch_fact(persist_t* p, uint64_t jobid, int state,
return rc;
}
/* bug-1687: WAL_JOB_START's own richer sibling of persist_submit_job_fact()
* -- mirrors persist_submit_job_dispatch_fact() above (bug-1678) exactly,
* but for the worker-confirmed-start fact: the sole durable record of the
* job's real job_pidinfo_t (pids/pgids/exdata a worker reported) and its
* true start_time. Payload shape: wal_jobstate_t header, immediately
* followed by a fixed uint64_t start_time (the receipt-time clock, captured
* ONCE by the caller rather than re-derived at each apply seam's own
* apply-time moment), immediately followed by the Cap'n Proto-encoded
* JobPIDInfo bytes -- apply_committed_entry()'s and apply_wal_entry()'s
* WAL_JOB_START cases decode exactly this shape, and treat a payload of
* exactly sizeof(wal_jobstate_t) (no trailing bytes) as "state-only,
* fabricate start_time and leave pidinfo untouched" for backward
* compatibility with any WAL_JOB_START entry written before this fix. */
int persist_submit_job_start_fact(persist_t* p, uint64_t jobid, int state,
uint64_t start_time,
const job_pidinfo_t* info)
{
wal_jobstate_t js = { .jobid = jobid, .state = (uint32_t)state,
._pad = 0 };
uint8_t* infobuf = NULL;
size_t infolen = 0;
uint8_t* payload;
size_t total;
int rc;
rc = capnp_encode_pidinfo(info, &infobuf, &infolen);
if (rc != 0)
return rc;
total = sizeof(js) + sizeof(start_time) + infolen;
payload = malloc(total);
if (!payload) { free(infobuf); return -ENOMEM; }
memcpy(payload, &js, sizeof(js));
memcpy(payload + sizeof(js), &start_time, sizeof(start_time));
memcpy(payload + sizeof(js) + sizeof(start_time), infobuf, infolen);
free(infobuf);
rc = wal_hook_append_classified(p, WAL_JOB_START, WAL_MODE_RETAIN,
payload, (uint32_t)total);
free(payload);
return rc;
}
/* bug-1679: see persist.h's doc comment for the full rationale. Mirrors
* apply_committed_entry()'s own identical Hazard-1-guarded hook fire
* (its shared WAL_JOB_START/.../DISPATCH case, right after its