---
id: "bug-1841"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-09-22T09:00:00Z"
last_seen: "2026-09-22T09:00:00Z"
occurrences: 1
files: ["src/controller/resync.c (resync_resolve_round() -- the fix)", "src/controller/persist.c/src/include/persist.h (new persist_notify_worker_status_updated()", "mirroring persist_notify_job_updated_already_persisted()/bug-1679)", "src/controller/testsuite/test_worker_resync.c (new regression test)"]
tags: ["auto-detected", "resource-leak", "resync", "worker-rejoin-resync", "resInUse", "scheduler", "controller", "hook-notification"]
related: ["bug-1679", "bug-1799", "bug-1801", "bug-1815", "bug-1812", "bug-1751"]
legacy: true
gitea_issue: 75
gitea_synced_status: "fixed"
---

# bug-1841: User-reported (xce_scheduler -i -s /tmp/scheduler-replay-20260922-071101.replay, worker pc13v, 22…

## Description
User-reported (xce_scheduler -i -s /tmp/scheduler-replay-20260922-071101.replay, worker pc13v, 22 real slots), also reported to occur in normal (non-sim) mode: after worker pc13v rejoined mid-replay (session~162, multiple WORKERS_PUSH+JOB_PUSH bursts, several `JOBS_PUSH: refusing resync state 0x1 for jobid ... existing job is RUN with a real allocation on record` WARNs -- bug-1812's guard correctly refusing a stale PEND resync), every one of that worker's jobs eventually legitimately finished (`info jobs` at session=423 shows all of jobids [0-22] DONE, [23-59] PEND, zero RUN), yet `info worker pc13v` still showed `resource slots: value=22.00 used=22.00` for the rest of the run (session 162 through 423, ~18 simulated minutes) -- the worker was permanently starved even though nothing was actually running on it.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
The worker-rejoin-resync feature's resync_arm_round() (resync.c) sets XCE_WORKER_RECONCILING on the controller's local worker_t.host.status and (via the HOOK_DETAIL_WORKER_JOIN hook already firing on rejoin) pushes it to the scheduler over WORKERS_PUSH. On the scheduler side, sched_reset_worker_()/sched_update_account() (bug-1799/bug-1801, 2026-09-11) deliberately FREEZE that worker's resInUse -- skip both the per-round zero-reset and the recharge -- while IS_WORKER_RECONCILING is set, by design, so in-flight reconciliation bookkeeping isn't clobbered mid-flight; bug-1799's own fix comment documents that normal reset+resum 'resumes and self-corrects on the very next round' once the status bit clears. But resync_resolve_round() (called from both the report-completion path, resync_apply_report -> resync_maybe_resolve, and the Req-5.3 timeout path, resync_on_timeout) only cleared XCE_WORKER_RECONCILING on the CONTROLLER's own local worker_t -- it never fired HOOK_DETAIL_WORKER_UPDATE or otherwise pushed the corrected status to the scheduler (contrast mark_worker_back()/worker.c:878-888, which does exactly that for the sibling XCE_WORKER_UNAVAIL bit). core_upsert_worker() (scheduler/src/core_events.c) only ever learns a worker's status from whatever WORKERS_PUSH it last received (`worker->status = w->status`, a bare unconditional copy, no other refresh path) -- so the scheduler's copy of the worker's status kept XCE_WORKER_RECONCILING set forever after the controller-side round resolved, permanently freezing resInUse at whatever it was when reconciliation started, even though every job on that worker had long since finished DONE. Reproduces identically in both sim/replay and live/normal mode since both drive the identical xce_core_begin_session() -> sched_reset_worker_()/sched_update_account() freeze logic; sim just replays whatever WORKERS_PUSH events the live controller actually sent, and the live controller never sent the correcting one either. test_worker_resync.c thoroughly tested that RECONCILING clears on the controller's OWN worker_t after resync_resolve_round(), but nothing tested that the scheduler-facing notification actually fires -- a gap noted separately during controller-peers-env-fallback design research (Sep 15 observation: 'full resync path lacks test coverage').

## Fix
FIXED (2026-09-22). Added persist_notify_worker_status_updated(cluster_t* c, worker_t* w) (persist.c, declared in persist.h next to persist_worker_hook_fun) -- the worker-side sibling of persist_notify_job_updated_already_persisted() (bug-1679): fires XCE_CONTROLLER_HOOK_WORKER's HOOK_DETAIL_WORKER_UPDATE AFTER_EVENT under the existing t_apply_seam_notifying guard so persist_worker_hook_fun (also subscribed to this hook, unconditionally WAL-appends on AFTER_EVENT) does NOT durably persist a WAL_WORKER_CTRL fact for this fire -- correct, since XCE_WORKER_RECONCILING/resync_state_t is intentionally ephemeral, re-derived fresh per rejoin, never replayed from WAL. resync_resolve_round() (resync.c) now calls this immediately after clearing XCE_WORKER_RECONCILING and detaching w->resync, guarded so it only fires when a round was genuinely outstanding (never on the already-documented idempotent no-op case). Verified safe under all three call sites' shared precondition (caller already holds w's write lock): confirmed neither HOOK_DETAIL_WORKER_UPDATE subscriber (sched_worker_change/apiserver_worker_change) nor persist_worker_hook_fun (guarded out entirely by the flag before touching w) locks w itself, so no deadlock risk. TDD: added test_resync_resolve_round_notifies_worker_update (test_worker_resync.c) -- a HOOK_DETAIL_WORKER_UPDATE spy hook proves resync_resolve_round() fires it exactly once per genuine resolve and not at all for the idempotent already-resolved no-op. RED confirmed first (Expected 1 Was 0) against the unfixed code, then GREEN after the fix. Full controller_unit_test suite: 1220 Tests, 0 Failures, 0 Ignored, OK (debug profile) -- no regressions. Note: the user's original .replay file is a frozen recording of the pre-fix controller's actual live traffic and will still show the frozen state when replayed, since sim mode only replays already-recorded events; the fix takes effect for new live runs/recordings going forward, not retroactively on old captures.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1679, bug-1799, bug-1801, bug-1815, bug-1812, bug-1751
- Gitea: #75