feat(worker-rejoin-resync): reconcile unrecognized job to pending (task 2.1)

Add record_job_reconcile_pending_fact(cluster_t*, job_t*) to job.c:
eagerly applies apply_job_update_transition(job, JOB_STAT_PEND) (real,
live mutation, not a deferred peek -- matching dispatch_a_job_'s
established pattern, not record_job_finish_fact's), releases
job->job.allocs on an applied (non-no-op) transition, then durably
appends via the generic persist_submit_job_fact(WAL_JOB_UPDATE). Caller
holds job's lock throughout and is responsible for calling
persist_notify_job_updated_already_persisted() after unlocking -- this
function deliberately does not call it itself.

Closes bug-1631 at this new call site: a RUN job with a real allocation
reconciled to PEND (Requirement 4.6, worker reports it unrecognized)
would otherwise carry a stale allocation into its next dispatch attempt,
exactly the gap bug-1631 documented as dormant until something re-admits
an already-dispatched job to the scheduler's PEND pool -- this feature
is that path. dispatch_a_job_ itself is unchanged (still correct for a
job's first-ever dispatch, the only case reachable there).

3 tests: allocs release + eager-mutation-before-unlock proof, no-op on
already-pending, and the Requirement 4.4 terminal-state-preservation
proof (rejects overwriting JOB_STAT_DONE).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
Rongsong Shen 2026-09-01 10:13:08 +00:00
parent eda2eb1bf0
commit cfe66dde16
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
7 changed files with 433 additions and 2 deletions

View file

@ -2866,6 +2866,71 @@ static int mark_job_update(cluster_t* c, job_t* job, int state)
return 0;
}
/* worker-rejoin-resync task 2.1 (Req 4.1-4.6, 5.4, 7.1-7.3): returns a job
* the worker reports as unrecognized back to PEND, releasing any resource
* allocation it was carrying so a subsequent redispatch attempt never
* inherits a stale allocation from a prior dispatch (bug-1631, confirmed
* live and deliberately left unfixed until this feature provided the first
* real re-admission path -- see design.md's Error Handling section).
*
* Follows dispatch_a_job_()'s own established pattern (job.c, its `if (rc
* >= 0) { ... }` success block): eager, synchronous mutation under the
* write lock the caller already holds, immediately followed by a durable
* WAL append, immediately followed (by the CALLER, after unlocking) by an
* explicit persist_notify_job_updated_already_persisted() call -- NOT
* record_job_finish_fact's/record_job_fail_fact's purely-deferred pattern
* (mutation withheld until apply_committed_entry replays the committed
* fact). The fact recorded here is something the worker already confirmed
* (or, in this case, failed to confirm) really happened -- ground truth
* already changed outside this controller -- which is exactly dispatch_a_
* job_'s own justification for going eager rather than deferred (design.md,
* "Controller: job.c" Components section).
*
* Precondition: caller holds `job` under RTTI_LOCK_WRITE on entry and keeps
* holding it until this call returns -- this function does not lock/unlock
* `job` itself, matching apply_job_update_transition's own calling
* convention (its other caller, mark_job_update above, is called under the
* exact same already-held write lock).
*
* apply_job_update_transition()'s existing job_state_is_backward guard is
* the ONLY commit/idempotency guard exercised here -- reused unchanged,
* never duplicated: a terminal job's transition is rejected
* (JOB_TRANSITION_REJECTED_BACKWARD, Req 4.4) and an already-PEND job's
* transition is a no-op (JOB_TRANSITION_NOOP_DUPLICATE, Req 4.3) -- neither
* outcome touches job->job.allocs or appends a WAL fact. Only on
* JOB_TRANSITION_APPLIED does this release job->job.allocs (if non-NULL)
* and durably append the fact via the existing generic persist_submit_job_
* fact() -- the plain {jobid, state} shape is sufficient here, same as
* every other WAL_JOB_UPDATE caller (no extra fields, unlike DISPATCH/
* START/FINISH's richer siblings).
*
* Returns the job_transition_result_t of the underlying transition, cast to
* int, so the caller can tell APPLIED apart from REJECTED_BACKWARD/
* NOOP_DUPLICATE. Unlike record_job_finish_fact/record_job_fail_fact, this
* function does NOT call persist_notify_job_updated_already_persisted --
* the caller must do so itself, after releasing the write lock, exactly
* once, for every JOB_TRANSITION_APPLIED result (cluster.h's doc comment on
* the prototype carries this same contract). Omitting it reproduces
* bug-1679 at this call site. */
int record_job_reconcile_pending_fact(cluster_t* c, job_t* job)
{
job_transition_result_t result;
result = apply_job_update_transition(job, JOB_STAT_PEND);
if (result != JOB_TRANSITION_APPLIED)
return (int)result;
if (job->job.allocs != NULL) {
free_JobAlloc_ptr(&(job->job.allocs));
job->job.allocs = NULL;
}
(void)persist_submit_job_fact(&c->persist, WAL_JOB_UPDATE, job->job.jobid,
JOB_STAT_PEND);
return (int)result;
}
/* Guarded state-transition core for a postdone fact, extracted from
* on_job_postdone's old inline mutation (raft-apply-decoupling task 6.1
* round 2, Req 3.3/7.1-7.3). Unlike a generic UPDATE, postdone's formula