apply_committed_entry() (persist.c), the single seam that applies every committed WAL entry, only ever fired the generic HOOK_DETAIL_JOB_UPDATE for job-state changes -- HOOK_DETAIL_JOB_START/JOB_EXIT were permanently dead code, since job.c only ever fires them at HOOK_BEFORE_EVENT (for WAL-append purposes). This left sched_job_start_ fun/sched_job_exit_fun (sched.c) and apiserver_job_start_fun/ apiserver_job_exit_fun (apiserver.c) unreachable, and the Go apiserver's already-wired OnJobFinished handler permanently starved of data -- a finished job's state could stay stuck (e.g. showing RUN) in the scheduler's own view and the apiserver never learned a job finished. The wire format was already enriched by bug-1697/bug-1689 with the pidinfo/exit_info data these hooks need, so this closes the gap rather than requiring new wire work. WAL_JOB_FINISH now fires HOOK_DETAIL_JOB_EXIT; WAL_JOB_START now fires HOOK_DETAIL_JOB_START, gated on a call-scoped decode flag rather than the transition result (WAL_JOB_START applies as a no-op duplicate for every real job in production per bug-1697, so gating on the transition result would make it dead code again). WAL_JOB_FAIL -> HOOK_DETAIL_JOB_FAIL has the same gap and is left as a follow-up. Also includes this investigation's supporting changes: a DEBUG3 log- retention bump (main.c, both controller and scheduler) so race-condition evidence survives a live burst without rotating out, and DEBUG3-gated diagnostic trace logging at the two points a job-update notification can fire or silently drop (job.c, sched.c). Live-verified end to end on a real deployed cluster: scheduler-side sched_job_state_change now fires for RUN and EXIT states, and the apiserver's OnJobFinished -- previously never invoked in production -- now correctly updates job status on completion. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA |
||
|---|---|---|
| .. | ||
| autoscaler | ||
| configs | ||
| controller | ||
| docs | ||
| frontend | ||
| include | ||
| lib | ||
| nixSupport | ||
| scheduler | ||
| schema | ||
| tools | ||
| worker | ||
| xceadmin | ||
| flake.lock | ||
| flake.nix | ||
| meson.build | ||
| meson.options | ||