wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(controller): mark RUN jobs unknown on worker loss and restart (bug-1509 P2)

Browse source 
Phase 2 of the job-finish robustness design (Part B, B1+B3).

B1 (worker liveness): a worker that stops reporting (heartbeat timeout,
already detected by check_worker_timeout_) or disconnects now has its RUN
jobs marked JOB_STAT_UNKWN|JOB_STAT_TRANSIENT (transient_since set, durable
WAL_JOB_UPDATE) instead of lingering as RUN forever. Marking happens
AFTER the worker lock is released (B5: avoids worker->jobmgr inversion
with the scheduler's jobmgr->worker order).

B3 (restart): persist_recover marks all recovered RUN jobs UNKWN+TRANSIENT
in-memory after replay — a controller restart leaves worker connectivity
unknown. In-memory only (no WAL append) because persist_recover may run
on a future standby that must never append.

Unit tests (red->green): worker-loss marks its RUN jobs unknown, other-host
jobs untouched, restart marks all RUN jobs. test_replay_apply_seam updated:
replay still drives facts to RUN, then B3 adds UNKWN+TRANSIENT.
This commit is contained in:
lei.chen 2026-08-06 05:57:59 +00:00
parent a793015f74
commit fc091348e8
7 changed files with 224 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

8
src/controller/include/cluster.h
View file

@ -50,6 +50,14 @@ extern int restore_job(cluster_t*, uint64_t, const void*, size_t, uint64_t);
* from apply_committed_entry's WAL_JOB_SUBMIT case after restore_job. */
extern int apply_registered_finish_if_any(cluster_t*, job_t*);
/* bug-1509 Phase 2 (B1): mark RUN jobs as UNKWN+TRANSIENT when their worker
* is lost (hostname set, durable WAL). Caller must not hold any
* worker/jobmgr lock (B5). */
extern void mark_worker_jobs_unknown(cluster_t*, const char* hostname);
/* bug-1509 Phase 2 (B3): after restart, mark ALL RUN jobs UNKWN+TRANSIENT
* in-memory only (no WAL — persist_recover may run on a future standby). */
extern void mark_all_run_jobs_unknown_in_memory(cluster_t*);
#ifdef XCE_TESTING
/* Direct probes of the bug-1509 unknown-finish registry (test binary only). */
extern int test_unknown_finish_register(const xce_jobfinish_t*);