wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(controller,worker): dual-state worker loss + resend actual job state (bug-1509 P3)

Browse source 
Phase 3 (B1 dual-state + B2): a job never lingers as RUN when its worker
is lost.

Dual-state B1 (user-approved): RDHUP (worker TCP dropped = confirmed dead)
marks its RUN jobs EXIT immediately (mark_worker_jobs_exit). Heartbeat
timeout (unknown) keeps them UNKWN+TRANSIENT awaiting reconnect
reconciliation; the TRANSIENT TTL GC is the backstop. handle_worker_leave
now calls mark_worker_jobs_exit (was mark_worker_jobs_unknown in Phase 2).

B2 (worker side): resend_job_status now checks the recovered process is
alive (kill(pid,0)) before re-reporting RUN — a dead process is reported
finished so the controller converges to EXIT instead of a stale RUN. And
resend_job_status runs on EVERY rejoin (checking_connection), not just
startup, so a reconnecting worker's jobs are reconciled (clears UNKWN via
the existing notify_jobstart claim path).

Design: docs/Job-Finish-Robustness-设计文档.md §4.4 (dual-state decision).

Unit test: worker RDHUP marks jobs EXIT (not UNKWN).
This commit is contained in:
lei.chen 2026-08-06 06:49:24 +00:00
parent fc091348e8
commit d3a92f1033
7 changed files with 135 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

6
src/worker/src/controller_connector.c
View file

@ -205,6 +205,12 @@ int checking_connection()
"Rejoined cluster successfully after retry %d",
retry_count);
/* bug-1509 Phase 3 (B2/B3): re-report running jobs on every
* rejoin, not just at startup, so the controller reconciles
* their state (clears UNKWN on a live claim; converges to a
* terminal state for processes that are gone). */
(void)resend_job_status(*cc_ctx->worker_list);
// api_ctx_t* api_ctx = (api_ctx_t*)xcecli_ctx;
// cc_ctx->xce_ctx = &(api_ctx->ctx);
return 0;

27
src/worker/src/xcecd_worker_utils.c
View file

@ -4,6 +4,7 @@
#include "xcecd_utils.h"
#include "xcelog.h"
#include <errno.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@ -451,19 +452,31 @@ bool resend_job_status(xcecd_worker_t* worker_list)
return true;
}
xcecd_job_t job = { .job_id = 0,
.job_pid = 0,
.job_status = JOB_RUNNING,
.exit_code = 0 };
xcecd_worker_t* worker = worker_list;
while (worker) {
if (worker->worker_status == WORKER_RUNNING) {
XCECD_LOG(XCE_LOG_DEBUG, "Resending job %lu status to controller.",
worker->job_info.jobid);
job.job_id = worker->job_info.jobid;
xcecd_reply_startjob(&job);
xcecd_job_t job;
memset(&job, 0, sizeof(job));
job.job_id = worker->job_info.jobid;
job.job_pid = worker->worker_pid;
/* bug-1509 B2: report the ACTUAL job state, not a blind RUN. The
* recovered worker's process may be gone (the worker crashed while
* the job was running) — in that case report it finished so the
* controller converges to a terminal state instead of a stale RUN. */
if (worker->worker_pid > 0 && kill(worker->worker_pid, 0) == 0) {
job.job_status = JOB_RUNNING;
job.stage = JOB_EXEC_STAGE_MAIN;
xcecd_reply_startjob(&job);
} else {
job.job_status = JOB_EXITED;
job.stage = JOB_EXEC_STAGE_MAIN;
job.exit_code = 1; /* process lost — terminal EXIT */
xcecd_job_finished(&job);
}
}
worker = worker->next;
}