wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(controller,worker): dual-state worker loss + resend actual job state (bug-1509 P3)

Browse source 
Phase 3 (B1 dual-state + B2): a job never lingers as RUN when its worker
is lost.

Dual-state B1 (user-approved): RDHUP (worker TCP dropped = confirmed dead)
marks its RUN jobs EXIT immediately (mark_worker_jobs_exit). Heartbeat
timeout (unknown) keeps them UNKWN+TRANSIENT awaiting reconnect
reconciliation; the TRANSIENT TTL GC is the backstop. handle_worker_leave
now calls mark_worker_jobs_exit (was mark_worker_jobs_unknown in Phase 2).

B2 (worker side): resend_job_status now checks the recovered process is
alive (kill(pid,0)) before re-reporting RUN — a dead process is reported
finished so the controller converges to EXIT instead of a stale RUN. And
resend_job_status runs on EVERY rejoin (checking_connection), not just
startup, so a reconnecting worker's jobs are reconciled (clears UNKWN via
the existing notify_jobstart claim path).

Design: docs/Job-Finish-Robustness-设计文档.md §4.4 (dual-state decision).

Unit test: worker RDHUP marks jobs EXIT (not UNKWN).
This commit is contained in:
lei.chen 2026-08-06 06:49:24 +00:00
parent fc091348e8
commit d3a92f1033
7 changed files with 135 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

4
src/controller/include/cluster.h
View file

@ -57,6 +57,10 @@ extern void mark_worker_jobs_unknown(cluster_t*, const char* hostname);
/* bug-1509 Phase 2 (B3): after restart, mark ALL RUN jobs UNKWN+TRANSIENT
* in-memory only (no WAL — persist_recover may run on a future standby). */
extern void mark_all_run_jobs_unknown_in_memory(cluster_t*);
/* bug-1509 Phase 3 (B1 dual-state): worker TCP dropped (RDHUP, confirmed dead)
* → mark its RUN jobs EXIT immediately. Caller must not hold any
* worker/jobmgr lock (B5). */
extern void mark_worker_jobs_exit(cluster_t*, const char* hostname);
#ifdef XCE_TESTING
/* Direct probes of the bug-1509 unknown-finish registry (test binary only). */