feat(worker-rejoin-resync): reconcile unqueried job assignment (task 2.2)
Add record_job_reconcile_assign_fact(cluster_t*, job_t*, worker_t*, xce_job_state_entry_t*) to job.c: eagerly sets job->job.state to match the worker's reported outcome (mirroring dispatch_a_job_'s inline mutation, minus allocation fields -- there are none to set, job->job.allocs stays NULL per Requirement 4.8's documented scope). Calls the existing persist_submit_job_dispatch_fact(..., alloc=NULL) directly, then leaves the post-unlock persist_notify_job_updated_already_persisted() call to the caller, same convention as task 2.1. Fixes bug-1746 along the way: capnp_encode_joballoc() (persist.c, part of bug-1678's original fix, predates this feature) unconditionally dereferenced a NULL alloc -- never triggered before since dispatch_a_job_ (the only prior caller) always passes non-NULL. Fixed with a minimal guard producing a true zero-length buffer, matching apply_committed_entry's alloc_len > 0 decode gate exactly. This directly contradicted design.md's "tested, backward-compatible on both seams" claim (true only for decode, never actually exercised for encode) -- corrected in 4 places. 2 tests: applies-running-no-allocs-and-notifies, and a real restart- replay round trip (write to real disk, fresh persist_recover(), confirm jobid/host/state/NULL-allocs survive) -- proving this task's own "durably survives a restart replay" completion criterion literally, not just the in-memory mutation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
parent
b7ed6de2a1
commit
ecfb4c942f
7 changed files with 550 additions and 3 deletions
|
|
@ -13,6 +13,7 @@
|
|||
#include "persist.h"
|
||||
#include "preason.h"
|
||||
#include "request.h"
|
||||
#include "typemap.h" /* xce_job_state_entry_t -- worker-rejoin-resync task 2.2 */
|
||||
#include "xce/cluster_def.h"
|
||||
#include "xce/errnox.h"
|
||||
#include "xce/types.h"
|
||||
|
|
@ -2931,6 +2932,124 @@ int record_job_reconcile_pending_fact(cluster_t* c, job_t* job)
|
|||
return (int)result;
|
||||
}
|
||||
|
||||
/* worker-rejoin-resync task 2.2 (Req 4.1, 4.3, 4.4, 4.7, 4.8, 7.1-7.3):
|
||||
* records a job the worker reports as running that the controller has no
|
||||
* existing non-terminal record of -- an "unqueried extra" (Req 3.6) the
|
||||
* controller never had a chance to ask about -- as assigned to the
|
||||
* reporting worker, through the same idempotent commit-then-apply path an
|
||||
* ordinary dispatch uses.
|
||||
*
|
||||
* Follows dispatch_a_job_()'s own established pattern (job.c, its `if (rc
|
||||
* >= 0) { ... }` success block, job.c:1686-1748) exactly as record_job_
|
||||
* reconcile_pending_fact above does: eager, synchronous mutation under the
|
||||
* write lock the caller already holds, immediately followed by a durable
|
||||
* WAL append, immediately followed (by the CALLER, after unlocking) by an
|
||||
* explicit persist_notify_job_updated_already_persisted() call -- NOT
|
||||
* record_job_finish_fact's/record_job_fail_fact's purely-deferred pattern.
|
||||
* The fact recorded here is something the worker already confirmed really
|
||||
* happened (it is running there right now) -- ground truth already changed
|
||||
* outside this controller -- exactly dispatch_a_job_'s own justification
|
||||
* for going eager rather than deferred (design.md, "Controller: job.c"
|
||||
* Components section).
|
||||
*
|
||||
* Precondition: caller holds `job` under RTTI_LOCK_WRITE on entry and keeps
|
||||
* holding it until this call returns -- this function does not lock/unlock
|
||||
* `job` itself, matching record_job_reconcile_pending_fact's own calling
|
||||
* convention above. Unlike that sibling, this function performs no
|
||||
* backward-transition guard check of its own: design.md's Service Interface
|
||||
* Invariants note that both new functions are "never called for a job the
|
||||
* controller already knows is terminal or more-final" as a PRECONDITION the
|
||||
* CALLER enforces, not something either function re-checks with its own
|
||||
* guard -- and for this specific outcome the precondition holds by
|
||||
* construction: Req 4.8's own trigger condition is precisely "the
|
||||
* Controller has no existing non-terminal record of that job assigned
|
||||
* elsewhere", so there is no terminal/more-final in-memory state this call
|
||||
* could possibly clobber. This mirrors dispatch_a_job_'s own inline
|
||||
* success-path mutation (job.c:1686-1717), which likewise sets
|
||||
* job->job.state directly, with no apply_job_update_transition/
|
||||
* job_state_is_backward call.
|
||||
*
|
||||
* "Host assignment": dispatch_a_job_'s own inline mutation (job.c:1686-
|
||||
* 1717) carries "which host" ENTIRELY via job->job.allocs -- the host name
|
||||
* lives inside xce_host_alloc_t.host, nested under xce_job_alloc_t.allocs.
|
||||
* Neither job_t nor xce_job_t (xce/types.h) has any OTHER field
|
||||
* representing a job's assigned worker/host independent of allocs
|
||||
* (confirmed by direct inspection of both struct definitions -- not a
|
||||
* guess). Since this function deliberately leaves job->job.allocs NULL (the
|
||||
* controller genuinely does not know what the scheduler originally computed
|
||||
* for this job -- design.md's Data Models "Documented scope decision"),
|
||||
* there is structurally no separate field left for this function to set to
|
||||
* represent "host assignment" beyond job->job.state itself. `w` is accepted
|
||||
* only to match design.md's own cited service-interface signature -- it is
|
||||
* otherwise inert here (see the (void)w; below): the still-running-outcome
|
||||
* check just below operates on `report->status`, not on `w`, and nothing on
|
||||
* job_t/xce_job_t is mutated from `w` either.
|
||||
*
|
||||
* Returns job_transition_result_t (cast to int), matching record_job_
|
||||
* reconcile_pending_fact's return-type convention above -- always
|
||||
* JOB_TRANSITION_APPLIED (this outcome has no reject/no-op case; see the
|
||||
* Precondition note). Unlike record_job_finish_fact/record_job_fail_fact,
|
||||
* this function does NOT call persist_notify_job_updated_already_persisted
|
||||
* -- the caller must do so itself, after releasing the write lock, exactly
|
||||
* once -- omitting it reproduces bug-1679 at this call site. */
|
||||
int record_job_reconcile_assign_fact(cluster_t* c, job_t* job, worker_t* w,
|
||||
xce_job_state_entry_t* report)
|
||||
{
|
||||
(void)w;
|
||||
|
||||
switch (report->status) {
|
||||
case JobStateStatus_running:
|
||||
break;
|
||||
default:
|
||||
/* This function's only defined caller usage (design.md's sequence
|
||||
* flow -- task 3.2, not yet implemented) is the "unqueried extra
|
||||
* job, still running" outcome: a finished extra routes through the
|
||||
* existing record_job_finish_fact/record_job_fail_fact instead
|
||||
* (Req 4.7), same as a queried finished job. Fail loud rather than
|
||||
* silently mis-recording a finished job as RUN, mirroring
|
||||
* dispatch_a_job_'s own "impossible per today's call convention"
|
||||
* ERROR+assert(0)-in-XCE_TESTING-then-safe-fallback idiom used
|
||||
* throughout this file. */
|
||||
ERROR("record_job_reconcile_assign_fact: called for job %lu with "
|
||||
"unexpected report status %d (expected JobStateStatus_running "
|
||||
"-- a finished/unrecognized outcome must route through a "
|
||||
"different fact function)",
|
||||
(unsigned long)job->job.jobid, report->status);
|
||||
#ifdef XCE_TESTING
|
||||
assert(0
|
||||
&& "record_job_reconcile_assign_fact called with a "
|
||||
"non-running report status");
|
||||
#endif
|
||||
break;
|
||||
}
|
||||
|
||||
job->job.state = JOB_STAT_RUN;
|
||||
|
||||
/* bug-1697: start_time is stamped HERE, synchronously alongside the RUN
|
||||
* transition, mirroring dispatch_a_job_'s own identical stamp
|
||||
* (job.c:1710) -- so that if this job's worker later also sends an
|
||||
* ordinary WAL_JOB_START report for it, that report always finds
|
||||
* state/start_time already set and applies as a harmless no-op, never
|
||||
* the genuine first RUN transition. There is no better time source: the
|
||||
* controller never observed this job's real dispatch time. */
|
||||
job->job.start_time = time(NULL);
|
||||
|
||||
/* Req 4.8 / design.md "Documented scope decision": job->job.allocs is
|
||||
* deliberately left untouched (NULL for a freshly-constructed job_t,
|
||||
* per this file's own makeJob()/create-job convention) -- the
|
||||
* controller has no way to learn what resource allocation the
|
||||
* scheduler originally computed for a job it never held any record of.
|
||||
* bug-1678's persist_submit_job_dispatch_fact() NULL-alloc case is
|
||||
* reused verbatim below, exactly as design.md specifies, rather than
|
||||
* fabricating one. */
|
||||
(void)persist_submit_job_dispatch_fact(&c->persist, job->job.jobid,
|
||||
JOB_STAT_RUN,
|
||||
(uint64_t)job->job.start_time,
|
||||
NULL);
|
||||
|
||||
return (int)JOB_TRANSITION_APPLIED;
|
||||
}
|
||||
|
||||
/* Guarded state-transition core for a postdone fact, extracted from
|
||||
* on_job_postdone's old inline mutation (raft-apply-decoupling task 6.1
|
||||
* round 2, Req 3.3/7.1-7.3). Unlike a generic UPDATE, postdone's formula
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue