lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): mark RUN jobs unknown on worker loss and restart (bug-1509 P2)

Browse source 
Phase 2 of the job-finish robustness design (Part B, B1+B3).

B1 (worker liveness): a worker that stops reporting (heartbeat timeout,
already detected by check_worker_timeout_) or disconnects now has its RUN
jobs marked JOB_STAT_UNKWN|JOB_STAT_TRANSIENT (transient_since set, durable
WAL_JOB_UPDATE) instead of lingering as RUN forever. Marking happens
AFTER the worker lock is released (B5: avoids worker->jobmgr inversion
with the scheduler's jobmgr->worker order).

B3 (restart): persist_recover marks all recovered RUN jobs UNKWN+TRANSIENT
in-memory after replay — a controller restart leaves worker connectivity
unknown. In-memory only (no WAL append) because persist_recover may run
on a future standby that must never append.

Unit tests (red->green): worker-loss marks its RUN jobs unknown, other-host
jobs untouched, restart marks all RUN jobs. test_replay_apply_seam updated:
replay still drives facts to RUN, then B3 adds UNKWN+TRANSIENT.
This commit is contained in:
lei.chen 2026-08-06 05:57:59 +00:00
parent a793015f74
commit fc091348e8
7 changed files with 224 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

12
src/controller/cluster.c
View file

@ -1731,7 +1731,9 @@ static int check_worker_timeout_(cluster_t* c, worker_t* w)
mark_worker_unavail(c, w);
return 0;
/* bug-1509 B1: return 1 so the caller (outside the worker lock) can
* mark this worker's RUN jobs as UNKWN+TRANSIENT (B5 lock order). */
return 1;
} else {
if (IS_WORKER_UNAVAIL(w->host.status)) {
mark_worker_back(c, w);
@ -1811,9 +1813,15 @@ static int check_workers_timeout(void* args)
}
lock(w, RTTI_LOCK_WRITE);
check_worker_timeout_(c, w);
int became_unavail = check_worker_timeout_(c, w);
unlock(w);
/* bug-1509 B1: worker timed out — mark its RUN jobs unknown AFTER
* releasing the worker lock (B5: avoids worker→jobmgr inversion). */
if (became_unavail) {
mark_worker_jobs_unknown(c, w->host.name);
}
dereference(w);
}