lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(bug-1737): scheduler can never learn a job finished successfully

Browse source 
sched_job_exit_fun() (renamed sched_job_finish_fun, sched.c) subscribes
to HOOK_DETAIL_JOB_EXIT, which fires for EVERY job finish, both success
and failure -- record_job_finish_fact() (job.c) decides DONE vs EXIT
from finish->exit_code BEFORE firing this same hook detail either way
(mirrors persist.c's own HOOK_DETAIL_JOB_EXIT -> WAL_JOB_FINISH
mapping). This function discarded info->exit_code and sent only a bare
jobid, forcing sched_handle_notify() (scheduler/src/handler.c) to
unconditionally map EVERY finish -- including a clean, successful one
-- to JOB_STAT_EXIT. A subsequent HOOK_DETAIL_JOB_UPDATE notify usually
self-corrected this moments later via the full job state, but if that
second notify was dropped during a scheduler reconnect gap (the same
window bug-1731/bug-1734 already documented), the scheduler was left
permanently stuck showing EXIT for a job the controller correctly
recorded as DONE.

Live-reproduced on a real 3-node HA cluster + worker: job
0-8-1788143749-0[38] genuinely completed successfully (worker log:
"completed successfully", controller joblist: DONE) but the
scheduler's own replay showed EXIT.

Fix: carry {jobid, exit_code} across the wire via the existing
JobFinishInfo shape (the same struct apiserver_job_exit_fun already
reuses for this identical hook) instead of a bare UInt64, so
sched_handle_notify() can make the identical exit_code==0-vs-not
decision job.c itself makes. Renamed sched_job_exit_fun to
sched_job_finish_fun to reflect what it actually does.

Live-verified: a 51-job run (all exit 0) showed 100% DONE on the
scheduler side after the fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
This commit is contained in:
Rongsong Shen 2026-08-31 04:20:27 +00:00
parent f01ae353ef
commit 026fe97b5a
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
6 changed files with 121 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

8
src/schema/controller.capnp
View file

@ -504,7 +504,13 @@ struct NotifyData $C.mapname("xce_notify_data_t"){
jobDispatchFail @2: JobFail $C.mapname("dispatchfail");
jobFinishInfo @3: JobFinishInfo $C.mapname("finishinfo");
schedStartJob @4: UInt64 $C.mapname("sched_start_job");
schedExitJob @5: UInt64 $C.mapname("sched_exit_job");
# bug-1737: was a bare UInt64 jobid -- carried no exit_code, so
# sched_handle_notify() (scheduler/src/handler.c) could not distinguish
# a successful finish (DONE) from a failed one (EXIT) and unconditionally
# assumed EXIT. Reuses the existing JobFinishInfo shape (jobid + code)
# apiserver_job_exit_fun (apiserver.c) already sends for this identical
# HOOK_DETAIL_JOB_EXIT hook, via NotifyData_data_jobFinishInfo.
schedExitJob @5: JobFinishInfo $C.mapname("sched_exit_job");
schedFailJob @6: UInt64 $C.mapname("sched_fail_job");
schedUpdateJob @7: SchedJob $C.mapname("sched_update_job");
schedUpdateResvals @8: ResValList $C.mapname("sched_update_resvals");