lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(bug-1737): scheduler can never learn a job finished successfully

Browse source 
sched_job_exit_fun() (renamed sched_job_finish_fun, sched.c) subscribes
to HOOK_DETAIL_JOB_EXIT, which fires for EVERY job finish, both success
and failure -- record_job_finish_fact() (job.c) decides DONE vs EXIT
from finish->exit_code BEFORE firing this same hook detail either way
(mirrors persist.c's own HOOK_DETAIL_JOB_EXIT -> WAL_JOB_FINISH
mapping). This function discarded info->exit_code and sent only a bare
jobid, forcing sched_handle_notify() (scheduler/src/handler.c) to
unconditionally map EVERY finish -- including a clean, successful one
-- to JOB_STAT_EXIT. A subsequent HOOK_DETAIL_JOB_UPDATE notify usually
self-corrected this moments later via the full job state, but if that
second notify was dropped during a scheduler reconnect gap (the same
window bug-1731/bug-1734 already documented), the scheduler was left
permanently stuck showing EXIT for a job the controller correctly
recorded as DONE.

Live-reproduced on a real 3-node HA cluster + worker: job
0-8-1788143749-0[38] genuinely completed successfully (worker log:
"completed successfully", controller joblist: DONE) but the
scheduler's own replay showed EXIT.

Fix: carry {jobid, exit_code} across the wire via the existing
JobFinishInfo shape (the same struct apiserver_job_exit_fun already
reuses for this identical hook) instead of a bare UInt64, so
sched_handle_notify() can make the identical exit_code==0-vs-not
decision job.c itself makes. Renamed sched_job_exit_fun to
sched_job_finish_fun to reflect what it actually does.

Live-verified: a 51-job run (all exit 0) showed 100% DONE on the
scheduler side after the fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
This commit is contained in:
Rongsong Shen 2026-08-31 04:20:27 +00:00
parent f01ae353ef
commit 026fe97b5a
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
6 changed files with 121 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

45
src/controller/sched.c
View file

@ -366,7 +366,30 @@ static int sched_job_start_fun(int type, int situation, void* data, va_list ap)
return rc;
}
static int sched_job_exit_fun(int type, int situation, void* data, va_list ap)
/*
* sched_job_finish_fun (bug-1737, renamed from sched_job_exit_fun) --
* subscribes to HOOK_DETAIL_JOB_EXIT, which despite its name fires for
* EVERY job finish, both success and failure (record_job_finish_fact(),
* job.c, decides DONE vs EXIT from finish->exit_code BEFORE firing this
* same hook detail either way -- mirrors persist.c's own WAL_JOB_FINISH ->
* HOOK_DETAIL_JOB_EXIT mapping, `case HOOK_DETAIL_JOB_EXIT: return
* WAL_JOB_FINISH;`). This function previously discarded info->exit_code and
* sent only the bare jobid, forcing sched_handle_notify() (scheduler/src/
* handler.c) to unconditionally map EVERY finish -- including a clean,
* successful one -- to JOB_STAT_EXIT. Under normal conditions a subsequent
* HOOK_DETAIL_JOB_UPDATE fire (persist.c, same WAL apply) self-corrected
* this moments later via schedUpdateJob's full xce_job_t (whose ->state is
* already the real terminal value), but if THAT second notify was dropped
* (c->sched.ep briefly NULL during a reconnect -- the same window bug-1731/
* bug-1734 already documented) the scheduler was left permanently stuck
* showing EXIT for a job the controller correctly recorded as DONE. Fix:
* carry info (jobid + exit_code) across the wire via the existing
* JobFinishInfo wire shape (same struct apiserver_job_exit_fun,
* apiserver.c, already reuses for this exact hook) so
* sched_handle_notify() can make the identical exit_code==0-vs-not decision
* job.c itself makes, instead of guessing EXIT unconditionally.
*/
static int sched_job_finish_fun(int type, int situation, void* data, va_list ap)
{
cluster_t* c = va_arg(ap, cluster_t*);
xce_jobfinish_t* info = va_arg(ap, xce_jobfinish_t*);
@ -375,7 +398,7 @@ static int sched_job_exit_fun(int type, int situation, void* data, va_list ap)
xce_notify_data_t xdata = {
.event = NotifyData_data_schedExitJob,
.data = {
.sched_exit_job = info->jobid
.sched_exit_job = info
}
};
/* clang-format on */
@ -396,6 +419,22 @@ static int sched_job_exit_fun(int type, int situation, void* data, va_list ap)
return rc;
}
/*
* sched_job_fail_fun -- subscribes to HOOK_DETAIL_JOB_FAIL, a narrower event
* than sched_job_finish_fun's HOOK_DETAIL_JOB_EXIT above: it fires for a
* PIPELINE-stage failure (dispatch/RPC error, or a post-exec script
* failing), not for the job's own exit code. apply_job_fail_transition()
* (job.c) only forces JOB_STAT_EXIT for the `default:` case (a genuinely
* pre-exit failure with no exit info yet); for JOB_EXEC_STAGE_POST_EXEC/
* Q_POST_EXEC it MERGES a JOB_STAT_PERR flag onto whatever DONE/EXIT the
* job's own finish already decided, never re-deciding DONE vs EXIT. This
* function still always sends the same bare-jobid "treat as EXIT" notify
* (matching the `default:` case only) -- bug-1738's
* job_state_regresses_terminal() guard in core_events.c's
* XCE_CORE_EVENT_JOB_STATE_CHANGE is what stops that from clobbering an
* already-DONE job for the POST_EXEC case, since the scheduler has no
* JOB_STAT_PERR-equivalent concept of its own to merge instead.
*/
static int sched_job_fail_fun(int type, int situation, void* data, va_list ap)
{
cluster_t* c = va_arg(ap, cluster_t*);
@ -514,7 +553,7 @@ static int sched_job_hook_fun(int type, int situation, int detail, void* data,
case HOOK_DETAIL_JOB_START:
return sched_job_start_fun(type, situation, data, ap);
case HOOK_DETAIL_JOB_EXIT:
return sched_job_exit_fun(type, situation, data, ap);
return sched_job_finish_fun(type, situation, data, ap);
case HOOK_DETAIL_JOB_UPDATE:
return sched_job_update_fun(type, situation, data, ap);
case HOOK_DETAIL_JOB_FAIL: