lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): resync_resolve_round never told scheduler RECONCILING cleared (bug-1841)

Browse source 
Worker-rejoin-resync sets XCE_WORKER_RECONCILING and pushes it to the
scheduler, which correctly freezes that worker's resInUse mid-reconciliation
(bug-1799/1801). But resync_resolve_round() only cleared the bit on the
controller's own worker_t, never notifying the scheduler -- so once a
rejoin round resolved, the scheduler's copy stayed RECONCILING forever,
permanently freezing that worker's resource accounting even after every
job on it finished. Live-observed via a replay capture: a 22-slot worker
stuck at used=22.00 for the rest of the run with zero RUN jobs left.

Added persist_notify_worker_status_updated() (mirrors bug-1679's job-side
persist_notify_job_updated_already_persisted()) and call it from
resync_resolve_round() so HOOK_DETAIL_WORKER_UPDATE fires without a
spurious WAL append. TDD: new spy-hook regression test, RED confirmed
against the unfixed code, GREEN after the fix. Full controller_unit_test
suite: 1220 Tests, 0 Failures, 0 Ignored.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-22 09:28:10 +00:00
parent e1da9f1e9b
commit 1f6f219da6
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
5 changed files with 147 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

17
src/include/persist.h
View file

@ -1527,6 +1527,23 @@ int persist_submit_job_finish_fact(persist_t* p, uint64_t jobid, int state,
* function is job.c's only sanctioned way to reach it. */
void persist_notify_job_updated_already_persisted(cluster_t* c, job_t* j);
/* bug-1841 (worker-side sibling of bug-1679/persist_notify_job_updated_
* already_persisted above): fires XCE_CONTROLLER_HOOK_WORKER's
* HOOK_DETAIL_WORKER_UPDATE AFTER_EVENT for `w`, guarded by the SAME
* t_apply_seam_notifying flag so persist_worker_hook_fun (also subscribed
* to this hook) does NOT append a fresh WAL_WORKER_CTRL fact for this fire.
* Unlike the job version, nothing here was "already persisted" -- this is
* for a worker mutation that is intentionally never durable at all (e.g.
* resync.c's XCE_WORKER_RECONCILING coordination bit, which is armed and
* cleared per rejoin and re-derived fresh every time, never replayed from
* WAL) but still needs HOOK_DETAIL_WORKER_UPDATE's live subscribers
* (sched_worker_change/apiserver_worker_change) to see `w`'s current
* status. Without this, a subscriber's only view of `w->host.status` is
* whatever the last WORKERS_PUSH-triggering hook fire carried -- resync.c's
* own resync_resolve_round() clearing the bit in memory is invisible to
* them until some unrelated worker event happens to re-fire the hook. */
void persist_notify_worker_status_updated(cluster_t* c, worker_t* w);
int persist_worker_hook_fun(int type, int situation, int detail,
void* cls, va_list ap);