lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller,worker): dual-state worker loss + resend actual job state (bug-1509 P3)

Browse source 
Phase 3 (B1 dual-state + B2): a job never lingers as RUN when its worker
is lost.

Dual-state B1 (user-approved): RDHUP (worker TCP dropped = confirmed dead)
marks its RUN jobs EXIT immediately (mark_worker_jobs_exit). Heartbeat
timeout (unknown) keeps them UNKWN+TRANSIENT awaiting reconnect
reconciliation; the TRANSIENT TTL GC is the backstop. handle_worker_leave
now calls mark_worker_jobs_exit (was mark_worker_jobs_unknown in Phase 2).

B2 (worker side): resend_job_status now checks the recovered process is
alive (kill(pid,0)) before re-reporting RUN — a dead process is reported
finished so the controller converges to EXIT instead of a stale RUN. And
resend_job_status runs on EVERY rejoin (checking_connection), not just
startup, so a reconnecting worker's jobs are reconciled (clears UNKWN via
the existing notify_jobstart claim path).

Design: docs/Job-Finish-Robustness-设计文档.md §4.4 (dual-state decision).

Unit test: worker RDHUP marks jobs EXIT (not UNKWN).
This commit is contained in:
lei.chen 2026-08-06 06:49:24 +00:00
parent fc091348e8
commit d3a92f1033
7 changed files with 135 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

26
src/controller/testsuite/test_apply_pump.c
View file

@ -1832,11 +1832,37 @@ static void test_restart_marks_all_run_jobs_unknown(void)
printf("test_restart_marks_all_run_jobs_unknown: PASS\n");
}
/* Phase 3 dual-state (B1): RDHUP (confirmed worker death) marks RUN jobs
* EXIT immediately — distinct from the heartbeat-timeout path (UNKWN). */
static void test_worker_rdhup_marks_jobs_exit(void)
{
tap_setup();
job_t* j = tap_seed_job(1007, JOB_STAT_RUN);
TEST_ASSERT_NOT_NULL(j);
lock(j, RTTI_LOCK_WRITE);
tap_set_job_allocs(j, "t1");
unlock(j);
mark_worker_jobs_exit(&g_tap_c, "t1");
TEST_ASSERT_TRUE_MESSAGE(j->job.state & JOB_STAT_EXIT,
"job must be EXIT after confirmed worker death");
TEST_ASSERT_FALSE_MESSAGE(j->job.state & JOB_STAT_UNKWN,
"confirmed-dead path is EXIT, not UNKWN");
TEST_ASSERT_TRUE_MESSAGE(j->job.finish_time > 0,
"finish_time must be set");
tap_teardown();
printf("test_worker_rdhup_marks_jobs_exit: PASS\n");
}
/* ── Test group entry point ───────────────────────────────────────────── */
void test_apply_pump(void)
{
UnitySetTestFile(__FILE__);
RUN_TEST(test_worker_rdhup_marks_jobs_exit);
RUN_TEST(test_worker_loss_marks_its_run_jobs_unknown);
RUN_TEST(test_worker_loss_does_not_touch_other_host_jobs);
RUN_TEST(test_restart_marks_all_run_jobs_unknown);