lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(bug-1731): fire HOOK_DETAIL_JOB_START/JOB_EXIT on WAL apply

Browse source 
apply_committed_entry() (persist.c), the single seam that applies every
committed WAL entry, only ever fired the generic HOOK_DETAIL_JOB_UPDATE
for job-state changes -- HOOK_DETAIL_JOB_START/JOB_EXIT were
permanently dead code, since job.c only ever fires them at
HOOK_BEFORE_EVENT (for WAL-append purposes). This left sched_job_start_
fun/sched_job_exit_fun (sched.c) and apiserver_job_start_fun/
apiserver_job_exit_fun (apiserver.c) unreachable, and the Go apiserver's
already-wired OnJobFinished handler permanently starved of data -- a
finished job's state could stay stuck (e.g. showing RUN) in the
scheduler's own view and the apiserver never learned a job finished.

The wire format was already enriched by bug-1697/bug-1689 with the
pidinfo/exit_info data these hooks need, so this closes the gap rather
than requiring new wire work. WAL_JOB_FINISH now fires
HOOK_DETAIL_JOB_EXIT; WAL_JOB_START now fires HOOK_DETAIL_JOB_START,
gated on a call-scoped decode flag rather than the transition result
(WAL_JOB_START applies as a no-op duplicate for every real job in
production per bug-1697, so gating on the transition result would make
it dead code again). WAL_JOB_FAIL -> HOOK_DETAIL_JOB_FAIL has the same
gap and is left as a follow-up.

Also includes this investigation's supporting changes: a DEBUG3 log-
retention bump (main.c, both controller and scheduler) so race-condition
evidence survives a live burst without rotating out, and DEBUG3-gated
diagnostic trace logging at the two points a job-update notification
can fire or silently drop (job.c, sched.c).

Live-verified end to end on a real deployed cluster: scheduler-side
sched_job_state_change now fires for RUN and EXIT states, and the
apiserver's OnJobFinished -- previously never invoked in production --
now correctly updates job status on completion.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
This commit is contained in:
Rongsong Shen 2026-08-29 11:29:10 +00:00
parent 42e1eeaa29
commit ebe691c3ed
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
9 changed files with 521 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

16
src/controller/testsuite/test_apply_reapply_idempotency.c
View file

@ -338,9 +338,11 @@ test_tari_finish_then_stale_start_preserves_terminal_state_via_dispatch(void)
sizeof(finish)));
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_DONE, j->job.state,
"FINISH must drive the job to its terminal state");
TEST_ASSERT_EQUAL_INT_MESSAGE(2, g_tari_hook_fires,
"FINISH must fire exactly one notification (SUBMIT's + FINISH's = "
"2 total so far)");
TEST_ASSERT_EQUAL_INT_MESSAGE(3, g_tari_hook_fires,
"bug-1731: a genuine forward WAL_JOB_FINISH transition now fires "
"TWO notifications (the pre-existing HOOK_DETAIL_JOB_UPDATE plus "
"the new HOOK_DETAIL_JOB_EXIT), not one -- SUBMIT's 1 + FINISH's "
"2 = 3 total so far; tari_spy_hook counts every AFTER_EVENT detail");
/* Step 3: apply a STALE, previously-unapplied START for the SAME job —
* simulating a duplicate/out-of-order fact arriving after the job
@ -361,9 +363,11 @@ test_tari_finish_then_stale_start_preserves_terminal_state_via_dispatch(void)
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_DONE, j->job.state,
"the job must remain in its FINISH-terminal state after the stale "
"START is applied");
TEST_ASSERT_EQUAL_INT_MESSAGE(2, g_tari_hook_fires,
"the stale, guard-rejected START must fire NO notification — the "
"count must stay exactly where FINISH left it");
TEST_ASSERT_EQUAL_INT_MESSAGE(3, g_tari_hook_fires,
"the stale, guard-rejected START must fire NO notification -- no "
"trailing pidinfo bytes means pidinfo_just_decoded stays false too "
"(bug-1731's HOOK_DETAIL_JOB_START gate), so the count must stay "
"exactly where FINISH left it (3)");
tari_teardown();
printf("test_tari_finish_then_stale_start_preserves_terminal_state_via_"