feat(worker-rejoin-resync): reconcile unqueried job assignment (task 2.2)

Add record_job_reconcile_assign_fact(cluster_t*, job_t*, worker_t*,
xce_job_state_entry_t*) to job.c: eagerly sets job->job.state to match
the worker's reported outcome (mirroring dispatch_a_job_'s inline
mutation, minus allocation fields -- there are none to set, job->job.allocs
stays NULL per Requirement 4.8's documented scope). Calls the existing
persist_submit_job_dispatch_fact(..., alloc=NULL) directly, then leaves
the post-unlock persist_notify_job_updated_already_persisted() call to
the caller, same convention as task 2.1.

Fixes bug-1746 along the way: capnp_encode_joballoc() (persist.c, part
of bug-1678's original fix, predates this feature) unconditionally
dereferenced a NULL alloc -- never triggered before since
dispatch_a_job_ (the only prior caller) always passes non-NULL. Fixed
with a minimal guard producing a true zero-length buffer, matching
apply_committed_entry's alloc_len > 0 decode gate exactly. This
directly contradicted design.md's "tested, backward-compatible on both
seams" claim (true only for decode, never actually exercised for
encode) -- corrected in 4 places.

2 tests: applies-running-no-allocs-and-notifies, and a real restart-
replay round trip (write to real disk, fresh persist_recover(), confirm
jobid/host/state/NULL-allocs survive) -- proving this task's own
"durably survives a restart replay" completion criterion literally, not
just the in-memory mutation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
Rongsong Shen 2026-09-01 11:40:47 +00:00
parent b7ed6de2a1
commit ecfb4c942f
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
7 changed files with 550 additions and 3 deletions

View file

@ -13,6 +13,7 @@
#include "persist.h"
#include "preason.h"
#include "request.h"
#include "typemap.h" /* xce_job_state_entry_t -- worker-rejoin-resync task 2.2 */
#include "xce/cluster_def.h"
#include "xce/errnox.h"
#include "xce/types.h"
@ -2931,6 +2932,124 @@ int record_job_reconcile_pending_fact(cluster_t* c, job_t* job)
return (int)result;
}
/* worker-rejoin-resync task 2.2 (Req 4.1, 4.3, 4.4, 4.7, 4.8, 7.1-7.3):
* records a job the worker reports as running that the controller has no
* existing non-terminal record of -- an "unqueried extra" (Req 3.6) the
* controller never had a chance to ask about -- as assigned to the
* reporting worker, through the same idempotent commit-then-apply path an
* ordinary dispatch uses.
*
* Follows dispatch_a_job_()'s own established pattern (job.c, its `if (rc
* >= 0) { ... }` success block, job.c:1686-1748) exactly as record_job_
* reconcile_pending_fact above does: eager, synchronous mutation under the
* write lock the caller already holds, immediately followed by a durable
* WAL append, immediately followed (by the CALLER, after unlocking) by an
* explicit persist_notify_job_updated_already_persisted() call -- NOT
* record_job_finish_fact's/record_job_fail_fact's purely-deferred pattern.
* The fact recorded here is something the worker already confirmed really
* happened (it is running there right now) -- ground truth already changed
* outside this controller -- exactly dispatch_a_job_'s own justification
* for going eager rather than deferred (design.md, "Controller: job.c"
* Components section).
*
* Precondition: caller holds `job` under RTTI_LOCK_WRITE on entry and keeps
* holding it until this call returns -- this function does not lock/unlock
* `job` itself, matching record_job_reconcile_pending_fact's own calling
* convention above. Unlike that sibling, this function performs no
* backward-transition guard check of its own: design.md's Service Interface
* Invariants note that both new functions are "never called for a job the
* controller already knows is terminal or more-final" as a PRECONDITION the
* CALLER enforces, not something either function re-checks with its own
* guard -- and for this specific outcome the precondition holds by
* construction: Req 4.8's own trigger condition is precisely "the
* Controller has no existing non-terminal record of that job assigned
* elsewhere", so there is no terminal/more-final in-memory state this call
* could possibly clobber. This mirrors dispatch_a_job_'s own inline
* success-path mutation (job.c:1686-1717), which likewise sets
* job->job.state directly, with no apply_job_update_transition/
* job_state_is_backward call.
*
* "Host assignment": dispatch_a_job_'s own inline mutation (job.c:1686-
* 1717) carries "which host" ENTIRELY via job->job.allocs -- the host name
* lives inside xce_host_alloc_t.host, nested under xce_job_alloc_t.allocs.
* Neither job_t nor xce_job_t (xce/types.h) has any OTHER field
* representing a job's assigned worker/host independent of allocs
* (confirmed by direct inspection of both struct definitions -- not a
* guess). Since this function deliberately leaves job->job.allocs NULL (the
* controller genuinely does not know what the scheduler originally computed
* for this job -- design.md's Data Models "Documented scope decision"),
* there is structurally no separate field left for this function to set to
* represent "host assignment" beyond job->job.state itself. `w` is accepted
* only to match design.md's own cited service-interface signature -- it is
* otherwise inert here (see the (void)w; below): the still-running-outcome
* check just below operates on `report->status`, not on `w`, and nothing on
* job_t/xce_job_t is mutated from `w` either.
*
* Returns job_transition_result_t (cast to int), matching record_job_
* reconcile_pending_fact's return-type convention above -- always
* JOB_TRANSITION_APPLIED (this outcome has no reject/no-op case; see the
* Precondition note). Unlike record_job_finish_fact/record_job_fail_fact,
* this function does NOT call persist_notify_job_updated_already_persisted
* -- the caller must do so itself, after releasing the write lock, exactly
* once -- omitting it reproduces bug-1679 at this call site. */
int record_job_reconcile_assign_fact(cluster_t* c, job_t* job, worker_t* w,
xce_job_state_entry_t* report)
{
(void)w;
switch (report->status) {
case JobStateStatus_running:
break;
default:
/* This function's only defined caller usage (design.md's sequence
* flow -- task 3.2, not yet implemented) is the "unqueried extra
* job, still running" outcome: a finished extra routes through the
* existing record_job_finish_fact/record_job_fail_fact instead
* (Req 4.7), same as a queried finished job. Fail loud rather than
* silently mis-recording a finished job as RUN, mirroring
* dispatch_a_job_'s own "impossible per today's call convention"
* ERROR+assert(0)-in-XCE_TESTING-then-safe-fallback idiom used
* throughout this file. */
ERROR("record_job_reconcile_assign_fact: called for job %lu with "
"unexpected report status %d (expected JobStateStatus_running "
"-- a finished/unrecognized outcome must route through a "
"different fact function)",
(unsigned long)job->job.jobid, report->status);
#ifdef XCE_TESTING
assert(0
&& "record_job_reconcile_assign_fact called with a "
"non-running report status");
#endif
break;
}
job->job.state = JOB_STAT_RUN;
/* bug-1697: start_time is stamped HERE, synchronously alongside the RUN
* transition, mirroring dispatch_a_job_'s own identical stamp
* (job.c:1710) -- so that if this job's worker later also sends an
* ordinary WAL_JOB_START report for it, that report always finds
* state/start_time already set and applies as a harmless no-op, never
* the genuine first RUN transition. There is no better time source: the
* controller never observed this job's real dispatch time. */
job->job.start_time = time(NULL);
/* Req 4.8 / design.md "Documented scope decision": job->job.allocs is
* deliberately left untouched (NULL for a freshly-constructed job_t,
* per this file's own makeJob()/create-job convention) -- the
* controller has no way to learn what resource allocation the
* scheduler originally computed for a job it never held any record of.
* bug-1678's persist_submit_job_dispatch_fact() NULL-alloc case is
* reused verbatim below, exactly as design.md specifies, rather than
* fabricating one. */
(void)persist_submit_job_dispatch_fact(&c->persist, job->job.jobid,
JOB_STAT_RUN,
(uint64_t)job->job.start_time,
NULL);
return (int)JOB_TRANSITION_APPLIED;
}
/* Guarded state-transition core for a postdone fact, extracted from
* on_job_postdone's old inline mutation (raft-apply-decoupling task 6.1
* round 2, Req 3.3/7.1-7.3). Unlike a generic UPDATE, postdone's formula