lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

merge: sync Rongsong's scheduler-framework (88 commits, f01ae353..c5b172c7)

Browse source 
Two-sided divergence from f01ae353 (2026-08-30): theirs +88 (worker
rejoin-resync reconciliation, terminal-state bug-1737/1738/1739/1764,
scheduler-sim verbosity, hermetic offline build, go mod vendor of Go deps
in apiserver -- 3375 pure-adds), ours +89 (job-finish robustness bug-1509
B1/B3, PEND-reason, worker reconnect resend, apiserver job control).

Conflict resolutions (kept both sides' fixes):
- controller_connector.c checking_connection(): combine theirs
  xcecd_resync_is_pending() rejoin deferral gate with ours
  resend_job_status() actual-state re-report on rejoin success.
- job.c: keep both unknown-finish registry (ours, bug-1509) and
  record_job_finish_fact_at (theirs, bug-1747); keep both
  mark_jobs_unknown_core family (ours) and record_job_reconcile_*
  (theirs). No double-notify: on_job_finish unchanged, theirs is
  resync-path-only.
- scheduler handler.c/test_handlers.c/meson: theirs schedExitJob
  exit_code branch (bug-1737) + ours jobDispatchFail success-ack no-op
  (bug-1512) + configpush enq_pend_update (bug-1513) both present; both
  test suites and REPL/sim targets wired.
- apiserver jobs.go: keep ours normalizeJobInt64/jobArrayIdx superset +
  JobIDHuman; drop theirs strict IntOrInt64 early-reject (superseded --
  ours tolerates float64/json.Number, no panic path).
- jobs_test.go: keep both regression suites (ours bug-1490, theirs
  bug-1742).
- .kiro/specs scheduler-sim-*/worker-rejoin-resync + .wolf/*: keep our
  deletions (established convention -- drop Rongsong bookkeeping on merge).

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-09-03 14:11:43 +00:00
commit 5e2985542f
3469 changed files with 1466599 additions and 382 deletions
Download patch file Download diff file Expand all files Collapse all files

302
src/controller/job.c
View file

@ -13,6 +13,7 @@
#include "persist.h"
#include "preason.h"
#include "request.h"
#include "typemap.h" /* xce_job_state_entry_t -- worker-rejoin-resync task 2.2 */
#include "xce/cluster_def.h"
#include "xce/errnox.h"
#include "xce/types.h"
@ -1509,7 +1510,66 @@ static int dispatch_a_job_(cluster_t* c, endpoint_t* ep, xce_job_alloc_t* alloc)
}
lock(w, RTTI_LOCK_READ);
if ((w->ep == NULL) || (w->host.status != 0)) {
if (w->ep == NULL) {
/* worker-rejoin-resync task 2.3 (Req 5.1, 5.4): a dispatch attempt
* to a worker the controller is not currently connected to (w->ep
* == NULL) is resolved here, BEFORE the w->host.status != 0 check
* below ever reaches xce_classify_dispatch_fail() at all --
* deliberately a bypass of that retry-then-terminal-fail machinery,
* not a third value added to xce_dispatch_fail_class_t (design.md's
* "Controller: job.c" Components section: that enum's whole
* contract is a two-way TERMINAL/RETRYABLE choice every existing
* call site's switch already exhaustively handles). Retrying the
* dispatch itself doesn't fit "the worker isn't even connected,"
* and this case must stay recoverable-via-reschedule rather than
* ever resolve to mark_job_fail()'s terminal outcome -- so this
* branch calls neither xce_classify_dispatch_fail() nor
* mark_job_fail(), and never touches j->dispatch_fail_count. */
int dfrc;
int result;
ERROR("worker %s is not connected (ep NULL), status %x",
w->host.name, w->host.status);
free_launchinfo(&launchinfo);
SET_ERROR(E_XCE_BAD_WORKER);
/* Capture errno immediately after SET_ERROR(), before any further
* call -- errno is not guaranteed stable across unlock(w)/
* dereference(w)/lock(j)/etc., mirroring the WORKER_NOT_READY
* branch's own discipline below. */
dfrc = -errno;
unlock(w);
dereference(w);
/* Req 5.4: resolves via record_job_reconcile_pending_fact() (task
* 2.1) directly -- eager mutation + allocs release (bug-1631) under
* this write lock, durable append inside that call, then (only on
* an actual APPLIED transition) this caller's own post-unlock
* persist_notify_job_updated_already_persisted() call, exactly as
* that function's own contract requires (cluster.h doc comment).
* Closes the same bug-1627 reference-leak class as the WORKER_NOT_
* READY branch below -- j's reference (held since the jobmgr lookup
* at the top of this function) is released via dereference(j) on
* every exit from this branch. */
lock(j, RTTI_LOCK_WRITE);
result = record_job_reconcile_pending_fact(c, j);
unlock(j);
if (result == (int)JOB_TRANSITION_APPLIED) {
persist_notify_job_updated_already_persisted(c, j);
}
/* JOB_TRANSITION_REJECTED_BACKWARD/NOOP_DUPLICATE: nothing changed
* (job already terminal, or already PEND) -- no notify needed,
* matching record_job_reconcile_pending_fact's own contract. */
dereference(j);
return dfrc;
} else if (w->host.status != 0) {
int dfrc;
xce_dispatch_fail_class_t dfclass;
job_failinfo_t failinfo = { .jobid = alloc->jobid };
@ -2803,6 +2863,64 @@ int test_unknown_finish_take(uint64_t jobid, xce_jobfinish_t* out)
}
#endif
/* worker-rejoin-resync task 3.2 (bug-1747 fix, Req 4.2): WAL_JOB_FINISH's
* own worker-reported-finish-time sibling of record_job_finish_fact() above
* -- identical guard/hook-fire/durable-append body, except it passes a
* CALLER-supplied finish time (converted from the worker's reported
* finish_time_ms, Req 3.5) to persist_submit_job_finish_fact() instead of
* always deriving time(NULL) at receipt time. record_job_finish_fact()
* itself is deliberately left UNMODIFIED: its own callers (on_job_finish's
* ordinary, near-instantaneous fire-and-forget worker report) are correctly
* served by receipt-time ~= true finish-time; only a RECONCILED finish
* (received long after the fact, on rejoin) needs the worker's own reported
* time instead -- mirroring task 2.1/2.2's own "new function alongside an
* unmodified existing one" pattern (record_job_reconcile_pending_fact/
* record_job_reconcile_assign_fact), not a new precedent.
*
* persist_submit_job_finish_fact()'s own `finish_time` parameter is SECONDS
* (time_t semantics) -- confirmed by its only other call site
* (record_job_finish_fact() above: `time_t finish_time = time(NULL);
* ...(uint64_t)finish_time`) and by the WAL_JOB_FINISH decode side
* (apply_committed_entry, persist.c: `j->job.finish_time =
* (time_t)wire_finish_time;`, a direct, unconverted assignment). The
* worker-reported wire field (xce_job_state_entry_t.finish_time_ms,
* typemap.h) is MILLISECONDS ("finishTimeMs" on the wire, Req 3.5) -- this
* function converts ms -> s (`finish_time_ms / 1000`) before calling
* persist_submit_job_finish_fact(), so the caller (resync.c) can pass the
* raw worker-reported value straight through without needing to know this
* unit detail itself.
*
* Same locking convention as record_job_finish_fact(): caller holds `job`
* under RTTI_LOCK_WRITE on entry and keeps holding it until this call
* returns; this function does not lock/unlock `job` itself. Same DEFERRED
* pattern too: durable append only here, no mutation -- the in-memory state
* change happens later via apply_committed_entry()'s replay, exactly as an
* ordinary finish already works (unlike record_job_reconcile_pending_fact/
* assign_fact's own EAGER pattern) -- so the caller must NOT call
* persist_notify_job_updated_already_persisted() after this returns; that
* notify happens later, at apply time, exactly as it already does for
* record_job_finish_fact(). */
int record_job_finish_fact_at(cluster_t* c, job_t* job,
xce_jobfinish_t* finish, uint64_t finish_time_ms)
{
if (job_state_is_terminal(job->job.state)) {
apply_job_finish_transition(job, finish); /* logs + no-op, unchanged */
return 0;
}
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
HOOK_DETAIL_JOB_EXIT, c, finish, job);
(void)persist_submit_job_finish_fact(&c->persist, job->job.jobid,
(finish->exit_code == 0)
? JOB_STAT_DONE
: JOB_STAT_EXIT,
finish_time_ms / 1000,
finish->exit_code);
return 0;
}
int on_job_finish(cluster_t* c, xce_jobfinish_t* finish)
{
job_t* job = NULL;
@ -3157,6 +3275,188 @@ void mark_worker_jobs_exit(cluster_t* c, const char* hostname)
}
iter_destroy(&it);
unlock(c->jobmgr);
/* worker-rejoin-resync task 2.1 (Req 4.1-4.6, 5.4, 7.1-7.3): returns a job
* the worker reports as unrecognized back to PEND, releasing any resource
* allocation it was carrying so a subsequent redispatch attempt never
* inherits a stale allocation from a prior dispatch (bug-1631, confirmed
* live and deliberately left unfixed until this feature provided the first
* real re-admission path -- see design.md's Error Handling section).
*
* Follows dispatch_a_job_()'s own established pattern (job.c, its `if (rc
* >= 0) { ... }` success block): eager, synchronous mutation under the
* write lock the caller already holds, immediately followed by a durable
* WAL append, immediately followed (by the CALLER, after unlocking) by an
* explicit persist_notify_job_updated_already_persisted() call -- NOT
* record_job_finish_fact's/record_job_fail_fact's purely-deferred pattern
* (mutation withheld until apply_committed_entry replays the committed
* fact). The fact recorded here is something the worker already confirmed
* (or, in this case, failed to confirm) really happened -- ground truth
* already changed outside this controller -- which is exactly dispatch_a_
* job_'s own justification for going eager rather than deferred (design.md,
* "Controller: job.c" Components section).
*
* Precondition: caller holds `job` under RTTI_LOCK_WRITE on entry and keeps
* holding it until this call returns -- this function does not lock/unlock
* `job` itself, matching apply_job_update_transition's own calling
* convention (its other caller, mark_job_update above, is called under the
* exact same already-held write lock).
*
* apply_job_update_transition()'s existing job_state_is_backward guard is
* the ONLY commit/idempotency guard exercised here -- reused unchanged,
* never duplicated: a terminal job's transition is rejected
* (JOB_TRANSITION_REJECTED_BACKWARD, Req 4.4) and an already-PEND job's
* transition is a no-op (JOB_TRANSITION_NOOP_DUPLICATE, Req 4.3) -- neither
* outcome touches job->job.allocs or appends a WAL fact. Only on
* JOB_TRANSITION_APPLIED does this release job->job.allocs (if non-NULL)
* and durably append the fact via the existing generic persist_submit_job_
* fact() -- the plain {jobid, state} shape is sufficient here, same as
* every other WAL_JOB_UPDATE caller (no extra fields, unlike DISPATCH/
* START/FINISH's richer siblings).
*
* Returns the job_transition_result_t of the underlying transition, cast to
* int, so the caller can tell APPLIED apart from REJECTED_BACKWARD/
* NOOP_DUPLICATE. Unlike record_job_finish_fact/record_job_fail_fact, this
* function does NOT call persist_notify_job_updated_already_persisted --
* the caller must do so itself, after releasing the write lock, exactly
* once, for every JOB_TRANSITION_APPLIED result (cluster.h's doc comment on
* the prototype carries this same contract). Omitting it reproduces
* bug-1679 at this call site. */
int record_job_reconcile_pending_fact(cluster_t* c, job_t* job)
{
job_transition_result_t result;
result = apply_job_update_transition(job, JOB_STAT_PEND);
if (result != JOB_TRANSITION_APPLIED)
return (int)result;
if (job->job.allocs != NULL) {
free_JobAlloc_ptr(&(job->job.allocs));
job->job.allocs = NULL;
}
(void)persist_submit_job_fact(&c->persist, WAL_JOB_UPDATE, job->job.jobid,
JOB_STAT_PEND);
return (int)result;
}
/* worker-rejoin-resync task 2.2 (Req 4.1, 4.3, 4.4, 4.7, 4.8, 7.1-7.3):
* records a job the worker reports as running that the controller has no
* existing non-terminal record of -- an "unqueried extra" (Req 3.6) the
* controller never had a chance to ask about -- as assigned to the
* reporting worker, through the same idempotent commit-then-apply path an
* ordinary dispatch uses.
*
* Follows dispatch_a_job_()'s own established pattern (job.c, its `if (rc
* >= 0) { ... }` success block, job.c:1686-1748) exactly as record_job_
* reconcile_pending_fact above does: eager, synchronous mutation under the
* write lock the caller already holds, immediately followed by a durable
* WAL append, immediately followed (by the CALLER, after unlocking) by an
* explicit persist_notify_job_updated_already_persisted() call -- NOT
* record_job_finish_fact's/record_job_fail_fact's purely-deferred pattern.
* The fact recorded here is something the worker already confirmed really
* happened (it is running there right now) -- ground truth already changed
* outside this controller -- exactly dispatch_a_job_'s own justification
* for going eager rather than deferred (design.md, "Controller: job.c"
* Components section).
*
* Precondition: caller holds `job` under RTTI_LOCK_WRITE on entry and keeps
* holding it until this call returns -- this function does not lock/unlock
* `job` itself, matching record_job_reconcile_pending_fact's own calling
* convention above. Unlike that sibling, this function performs no
* backward-transition guard check of its own: design.md's Service Interface
* Invariants note that both new functions are "never called for a job the
* controller already knows is terminal or more-final" as a PRECONDITION the
* CALLER enforces, not something either function re-checks with its own
* guard -- and for this specific outcome the precondition holds by
* construction: Req 4.8's own trigger condition is precisely "the
* Controller has no existing non-terminal record of that job assigned
* elsewhere", so there is no terminal/more-final in-memory state this call
* could possibly clobber. This mirrors dispatch_a_job_'s own inline
* success-path mutation (job.c:1686-1717), which likewise sets
* job->job.state directly, with no apply_job_update_transition/
* job_state_is_backward call.
*
* "Host assignment": dispatch_a_job_'s own inline mutation (job.c:1686-
* 1717) carries "which host" ENTIRELY via job->job.allocs -- the host name
* lives inside xce_host_alloc_t.host, nested under xce_job_alloc_t.allocs.
* Neither job_t nor xce_job_t (xce/types.h) has any OTHER field
* representing a job's assigned worker/host independent of allocs
* (confirmed by direct inspection of both struct definitions -- not a
* guess). Since this function deliberately leaves job->job.allocs NULL (the
* controller genuinely does not know what the scheduler originally computed
* for this job -- design.md's Data Models "Documented scope decision"),
* there is structurally no separate field left for this function to set to
* represent "host assignment" beyond job->job.state itself. `w` is accepted
* only to match design.md's own cited service-interface signature -- it is
* otherwise inert here (see the (void)w; below): the still-running-outcome
* check just below operates on `report->status`, not on `w`, and nothing on
* job_t/xce_job_t is mutated from `w` either.
*
* Returns job_transition_result_t (cast to int), matching record_job_
* reconcile_pending_fact's return-type convention above -- always
* JOB_TRANSITION_APPLIED (this outcome has no reject/no-op case; see the
* Precondition note). Unlike record_job_finish_fact/record_job_fail_fact,
* this function does NOT call persist_notify_job_updated_already_persisted
* -- the caller must do so itself, after releasing the write lock, exactly
* once -- omitting it reproduces bug-1679 at this call site. */
int record_job_reconcile_assign_fact(cluster_t* c, job_t* job, worker_t* w,
xce_job_state_entry_t* report)
{
(void)w;
switch (report->status) {
case JobStateStatus_running:
break;
default:
/* This function's only defined caller usage (design.md's sequence
* flow -- task 3.2, not yet implemented) is the "unqueried extra
* job, still running" outcome: a finished extra routes through the
* existing record_job_finish_fact/record_job_fail_fact instead
* (Req 4.7), same as a queried finished job. Fail loud rather than
* silently mis-recording a finished job as RUN, mirroring
* dispatch_a_job_'s own "impossible per today's call convention"
* ERROR+assert(0)-in-XCE_TESTING-then-safe-fallback idiom used
* throughout this file. */
ERROR("record_job_reconcile_assign_fact: called for job %lu with "
"unexpected report status %d (expected JobStateStatus_running "
"-- a finished/unrecognized outcome must route through a "
"different fact function)",
(unsigned long)job->job.jobid, report->status);
#ifdef XCE_TESTING
assert(0
&& "record_job_reconcile_assign_fact called with a "
"non-running report status");
#endif
break;
}
job->job.state = JOB_STAT_RUN;
/* bug-1697: start_time is stamped HERE, synchronously alongside the RUN
* transition, mirroring dispatch_a_job_'s own identical stamp
* (job.c:1710) -- so that if this job's worker later also sends an
* ordinary WAL_JOB_START report for it, that report always finds
* state/start_time already set and applies as a harmless no-op, never
* the genuine first RUN transition. There is no better time source: the
* controller never observed this job's real dispatch time. */
job->job.start_time = time(NULL);
/* Req 4.8 / design.md "Documented scope decision": job->job.allocs is
* deliberately left untouched (NULL for a freshly-constructed job_t,
* per this file's own makeJob()/create-job convention) -- the
* controller has no way to learn what resource allocation the
* scheduler originally computed for a job it never held any record of.
* bug-1678's persist_submit_job_dispatch_fact() NULL-alloc case is
* reused verbatim below, exactly as design.md specifies, rather than
* fabricating one. */
(void)persist_submit_job_dispatch_fact(&c->persist, job->job.jobid,
JOB_STAT_RUN,
(uint64_t)job->job.start_time,
NULL);
return (int)JOB_TRANSITION_APPLIED;
}
/* Guarded state-transition core for a postdone fact, extracted from