lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(scheduler-dispatch-commit): controller resolves terminal dispatch failures to EXIT (task 3.2)

Browse source 
The two terminal-classified failure sites in dispatch_a_job_() (invalid
allocation, queue-exec-info lookup failure) now actually resolve the job
instead of leaving it stuck at PEND forever with a dangling reference to
nowhere. Each site: lock(j, RTTI_LOCK_WRITE) -> genuinely calls
xce_classify_dispatch_fail(site) and branches on the result (not a
hardcoded assumption) -> mark_job_fail() under the lock (stage=0 routes
apply_job_fail_transition() to JOB_STAT_EXIT) -> unlock(j) -> the existing
dereference(j) (kept exactly where it was, not duplicated) -> the existing
return code, unchanged. No notify_dispatch() added -- mark_job_fail()'s
own hook chain is the sole notification for a terminal outcome per
design.md; handle_jobdispatch()'s still-unconditional notify_dispatch()
call is untouched (task 4.1's job).

New test_dispatch_fail_terminal.c proves both sites resolve to
JOB_STAT_EXIT with allocs left NULL, using a canary-reference technique
to prove no reference over-release (job_t's real refcount has no test
accessor). Independently reviewed and approved; applied the reviewer's
one non-blocking fast-follow (arm the canary before the call under test,
not after, so it's an independent proof rather than relying on other
assertions to crash first) and re-verified with a clean incremental build.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
This commit is contained in:
Rongsong Shen 2026-08-23 16:36:39 +00:00
parent d9aedc4b34
commit 8a4266ada4
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
5 changed files with 436 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

102
src/controller/job.c
View file

@ -958,6 +958,54 @@ static int dispatch_a_job_(cluster_t* c, endpoint_t* ep, xce_job_alloc_t* alloc)
nc = alloc->nhosts;
if (nc <= 0) {
/* scheduler-dispatch-commit task 3.2 (Req 4.2, 4.5): resolves this
* terminal (locally-invalid) dispatch failure -- j's own reference
* held since the lookup above is released via the dereference(j)
* call below exactly as it already was before this task (not
* duplicated). Nothing was ever committed to j->job.allocs on this
* path (task 3.1 gates that commit on a successful
* xce_startjob_req(), never reached here), so Req 4.5's "release
* any tentative allocation data" is satisfied vacuously by
* construction -- there is nothing to release. */
xce_dispatch_fail_class_t dfclass;
job_failinfo_t failinfo = { .jobid = alloc->jobid };
lock(j, RTTI_LOCK_WRITE);
dfclass
= xce_classify_dispatch_fail(XCE_DISPATCH_FAIL_SITE_INVALID_ALLOC);
switch (dfclass) {
case XCE_DISPATCH_FAIL_TERMINAL:
break;
case XCE_DISPATCH_FAIL_RETRYABLE:
default:
/* Impossible per today's classification table (confirmed:
* xce_classify_dispatch_fail() returns TERMINAL for this site
* unconditionally) -- task 3.3 is what wires real retryable
* handling at the other four sites, not this one. Fail loud
* and fall through to the same terminal resolution as a safe
* fallback, mirroring xce_classify_dispatch_fail()'s own
* default: convention. */
ERROR("dispatch_a_job_: XCE_DISPATCH_FAIL_SITE_INVALID_ALLOC "
"classified non-terminal (%d) -- impossible per the "
"current classification table; resolving terminal as a "
"safe fallback", (int)dfclass);
#ifdef XCE_TESTING
assert(0
&& "XCE_DISPATCH_FAIL_SITE_INVALID_ALLOC classified "
"non-terminal -- update the classification table or "
"this call site");
#endif
break;
}
failinfo.code = -EINVAL;
failinfo.detail = "dispatch failed: invalid allocation";
mark_job_fail(c, j, &failinfo);
unlock(j);
dereference(j);
return -EINVAL;
@ -1007,8 +1055,62 @@ static int dispatch_a_job_(cluster_t* c, endpoint_t* ep, xce_job_alloc_t* alloc)
rc = get_queue_exec(c, j->job.submit->queue, &q_pre_exec, &q_post_exec);
if (rc < 0) {
/* scheduler-dispatch-commit task 3.2 (Req 4.2, 4.5): resolves
* this terminal (locally-invalid) dispatch failure -- j's own
* reference held since the lookup above is released via the
* dereference(j) call below exactly as it already was before
* this task (not duplicated). Nothing was ever committed to
* j->job.allocs on this path (task 3.1 gates that commit on a
* successful xce_startjob_req(), never reached here), so Req
* 4.5's "release any tentative allocation data" is satisfied
* vacuously by construction. unlock(w)/dereference(w)/
* free_launchinfo(&launchinfo) stay in their existing
* positions -- only this new lock/classify/mark_job_fail/
* unlock sequence is inserted around the existing
* dereference(j) call. */
xce_dispatch_fail_class_t dfclass;
job_failinfo_t failinfo = { .jobid = alloc->jobid };
unlock(w);
dereference(w);
lock(j, RTTI_LOCK_WRITE);
dfclass = xce_classify_dispatch_fail(
XCE_DISPATCH_FAIL_SITE_QUEUE_EXEC);
switch (dfclass) {
case XCE_DISPATCH_FAIL_TERMINAL:
break;
case XCE_DISPATCH_FAIL_RETRYABLE:
default:
/* Impossible per today's classification table (confirmed:
* xce_classify_dispatch_fail() returns TERMINAL for this
* site unconditionally) -- task 3.3 is what wires real
* retryable handling at the other four sites, not this
* one. Fail loud and fall through to the same terminal
* resolution as a safe fallback, mirroring
* xce_classify_dispatch_fail()'s own default: convention. */
ERROR("dispatch_a_job_: XCE_DISPATCH_FAIL_SITE_QUEUE_EXEC "
"classified non-terminal (%d) -- impossible per the "
"current classification table; resolving terminal "
"as a safe fallback", (int)dfclass);
#ifdef XCE_TESTING
assert(0
&& "XCE_DISPATCH_FAIL_SITE_QUEUE_EXEC classified "
"non-terminal -- update the classification table "
"or this call site");
#endif
break;
}
failinfo.code = rc;
failinfo.detail = "dispatch failed: queue exec info lookup "
"failed";
mark_job_fail(c, j, &failinfo);
unlock(j);
dereference(j);
free_launchinfo(&launchinfo);