lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): persist dispatch-time resource allocation into WAL_JOB_DISPATCH (bug-1678, gitea #43)

Browse source 
dispatch_a_job_() set j->job.allocs in memory at dispatch-commit time
but only ever persisted {jobid, state} via persist_submit_job_fact() --
the allocation itself was never written to the WAL. On a full restart,
job.state correctly recovered to RUN (both WAL_JOB_DISPATCH and the
later WAL_JOB_START fact carry state=RUN) but job.allocs stayed
permanently NULL, since apply_job_update_transition() only ever
touches state. EXEC_HOST displayed blank forever for a job that was
genuinely running before shutdown.

No capnp schema change needed -- JobAlloc (xce_job_alloc_t) was
already a full top-level struct (the existing Job.allocs field) with
generated encode/decode functions already in the build. New:
capnp_encode_joballoc()/decode_joballoc() (persist.c, mirroring the
existing capnp_encode_worker()/restore_job() patterns) and
persist_submit_job_dispatch_fact() -- WAL_JOB_DISPATCH's payload is
now a wal_jobstate_t header plus trailing capnp-encoded JobAlloc
bytes. dispatch_a_job_() now persists j->job.allocs (the canonical,
freshly-constructed struct, read under the same write lock) instead
of just the state.

Both apply seams updated: apply_committed_entry() (primary's own boot
replay) gets one added branch in its existing WAL_JOB_START/FINISH
stamp-chain; apply_wal_entry() (standby catch-up / replicated-apply --
also required, a promoted standby would hit the identical bug) has
WAL_JOB_DISPATCH split into its own case, calling new
restore_job_dispatch_state(). A payload of exactly sizeof(wal_jobstate_t)
(any entry written before this fix) decodes no allocation and behaves
exactly as before -- full backward compatibility.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 03:12:49 +00:00
parent 2a24551542
commit db142d415f
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
4 changed files with 205 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

46
src/controller/job.c
View file

@ -785,6 +785,36 @@ int restore_job_state(cluster_t* c, uint64_t jobid, uint32_t state)
return 0;
}
/* bug-1678: WAL_JOB_DISPATCH's own sibling of restore_job_state() above, for
* apply_wal_entry()'s standby catch-up / replicated-apply seam -- takes
* ownership of an optional decoded resource allocation (NULL for a
* state-only WAL_JOB_DISPATCH entry, e.g. one written before this fix) so a
* promoted standby recovers EXEC_HOST the same way apply_committed_entry's
* WAL_JOB_DISPATCH case already does for the primary's own boot replay. */
int restore_job_dispatch_state(cluster_t* c, uint64_t jobid, uint32_t state,
xce_job_alloc_t* alloc)
{
job_t* j = NULL;
lock(c->jobmgr, RTTI_LOCK_READ);
key_find(c->jobmgr, (void*)jobid, (void**)&j);
unlock(c->jobmgr);
if (!j) {
if (alloc != NULL)
free_JobAlloc_ptr(&alloc);
return -ENOENT;
}
lock(j, RTTI_LOCK_WRITE);
j->job.state = (int)state;
if (alloc != NULL) {
if (j->job.allocs != NULL)
free_JobAlloc_ptr(&(j->job.allocs));
j->job.allocs = alloc;
}
unlock(j);
return 0;
}
static int get_nslots_in_alloc(xce_host_alloc_t* alloc)
{
int i;
@ -1362,12 +1392,16 @@ static int dispatch_a_job_(cluster_t* c, endpoint_t* ep, xce_job_alloc_t* alloc)
j->dispatch_fail_count = 0;
/* Durable record of the dispatch-time RUN transition (task 1.2),
* mirroring record_job_start_fact()'s persist_submit_job_fact()
* call shape exactly -- _pad stays 0 (the "legacy: state is
* absolute value" convention, matching every other job-state fact
* committed synchronously here rather than via deferred apply). */
(void)persist_submit_job_fact(&c->persist, WAL_JOB_DISPATCH,
alloc->jobid, JOB_STAT_RUN);
* now including the resource allocation itself (bug-1678): the
* plain {jobid, state} shape persist_submit_job_fact() writes was
* not enough to reconstruct job.allocs on a restart replay, leaving
* EXEC_HOST permanently blank for a job that was RUNNING before
* shutdown. j->job.allocs is read here, still under this same
* write lock, as the freshly-constructed canonical struct just
* stored above -- not the (now partially moved-from) local
* `alloc`/`allocs` variables. */
(void)persist_submit_job_dispatch_fact(&c->persist, alloc->jobid,
JOB_STAT_RUN, j->job.allocs);
unlock(j);
} else {