lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): resync_resolve_round never told scheduler RECONCILING cleared (bug-1841)

Browse source 
Worker-rejoin-resync sets XCE_WORKER_RECONCILING and pushes it to the
scheduler, which correctly freezes that worker's resInUse mid-reconciliation
(bug-1799/1801). But resync_resolve_round() only cleared the bit on the
controller's own worker_t, never notifying the scheduler -- so once a
rejoin round resolved, the scheduler's copy stayed RECONCILING forever,
permanently freezing that worker's resource accounting even after every
job on it finished. Live-observed via a replay capture: a 22-slot worker
stuck at used=22.00 for the rest of the run with zero RUN jobs left.

Added persist_notify_worker_status_updated() (mirrors bug-1679's job-side
persist_notify_job_updated_already_persisted()) and call it from
resync_resolve_round() so HOOK_DETAIL_WORKER_UPDATE fires without a
spurious WAL append. TDD: new spy-hook regression test, RED confirmed
against the unfixed code, GREEN after the fix. Full controller_unit_test
suite: 1220 Tests, 0 Failures, 0 Ignored.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-22 09:28:10 +00:00
parent e1da9f1e9b
commit 1f6f219da6
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
5 changed files with 147 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

21
src/controller/resync.c
View file

@ -393,11 +393,6 @@ int resync_arm_round(cluster_t* c, worker_t* w)
void resync_resolve_round(cluster_t* c, worker_t* w)
{
(void)c; /* accepted to match this file's other functions' shape and
* what a later caller (task 3.2) already has in hand
* alongside `w` -- this function's own body has nothing to do
* with `c` today. */
if (w == NULL || w->resync == NULL)
return;
@ -406,6 +401,22 @@ void resync_resolve_round(cluster_t* c, worker_t* w)
free(w->resync->queried_jobids);
free(w->resync);
w->resync = NULL;
/* bug-1841: every call site here (resync_on_timeout, resync_maybe_
* resolve via resync_settle_entry, and resync_arm_round's own "discard
* a still-outstanding prior round" branch) already holds w's write
* lock, matching this function's documented precondition -- and
* persist_notify_worker_status_updated()'s only two live subscribers
* (sched_worker_change/apiserver_worker_change) never lock `w`
* themselves, so firing it here is safe. Without this, the scheduler's
* own copy of `w`'s status (fed purely from whatever WORKERS_PUSH it
* last received) keeps XCE_WORKER_RECONCILING set forever once this
* clears it only in the controller's local worker_t -- permanently
* freezing that worker's resInUse (bug-1799/1801's by-design freeze)
* even after every job on it has finished. */
if (c != NULL) {
persist_notify_worker_status_updated(c, w);
}
}
int resync_round_is_current(worker_t* w, uint64_t token)