lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693)

Browse source 
Violates this project's own recorded Decision Log entry: "No client
reply-wait timeout as a fix for a hung request -- fail loud at the real
transport/source instead of masking with a deadline backstop."

OPCode_schedAdmin (cluster.c) and on_fetchfile (job.c, the worker proxy
-- cluster.c's own comment says schedAdmin "mirrors on_fetchfile's proxy
pattern", confirming one shared design mistake, not two independent
bugs) both blocked on a multi-second pthread_cond_timedwait, replying
"timeout" on expiry. handle_scheduler_leave()/handle_worker_leave() (the
actual disconnect handlers) only nulled the endpoint pointer -- neither
ever signaled the pending waiter, so a mid-wait disconnect was invisible
until the deadline lapsed. The timeout was the ONLY failure-detection
mechanism, exactly what the decision forbids.

Both sites already had a working signal function on their success path
(schedadmin_signal(), fetchfile_signal()) -- just never wired to the
disconnect handlers. Fixed by calling them from
handle_scheduler_leave()/handle_worker_leave() with a distinguishable
-ECONNRESET outcome, so the waiter wakes immediately on disconnect
instead of only on the deadline. Timeouts kept as defensive backstops
for a genuinely wedged peer that never disconnects cleanly either --
this fix is about not relying SOLELY on the timeout, not removing it.

schedadmin_signal_disconnect() must be called BEFORE handle_scheduler_leave()
acquires c->sched.lock -- the dispatch wait holds that same lock for its
entire duration, so signaling after acquiring it would deadlock.

Found by an /xce-audit-invariants controller pass (Decision Log fidelity
dimension). New test_fetchfile_disconnect.c plus extensions to
test_sched_admin_dispatch.c/test_sched_worker_change.c; RED/GREEN
independently verified against real elapsed-time assertions (pre-fix:
5001ms full-timeout wait; post-fix: <1s immediate wake). Full controller
suite: 1181 tests, 2 failures -- both the pre-existing, unrelated
bug-1561 overlayfs sandbox flake, confirmed via a clean meson-driven run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 14:13:26 +00:00
parent c83b388be2
commit 3eba7c16bf
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
12 changed files with 1032 additions and 424 deletions
Download patch file Download diff file Expand all files Collapse all files

83
src/controller/job.c
View file

@ -3307,6 +3307,11 @@ int do_clr_preason(cluster_t* c, endpoint_t* ep, xce_request_t* request)
typedef struct fetchfile_pending {
uint64_t jobid;
worker_t* worker; /* bug-1693: which worker this request was
* proxied to -- lets a disconnect be
* matched to the pending waiter(s) it
* affects; see
* fetchfile_signal_worker_disconnect(). */
pthread_mutex_t mu;
pthread_cond_t cv;
int done;
@ -3364,6 +3369,78 @@ void fetchfile_signal(uint64_t jobid, int status, xce_fetchfile_reply_t* reply)
pthread_mutex_unlock(&p->mu);
}
/* bug-1693 (Decision Log: "No client reply-wait timeout as a fix for a hung
* request -- fail loud at the real transport/source instead of masking with
* a deadline backstop"). Before this fix, handle_worker_leave() (worker.c)
* only ever nulled w->ep on a worker disconnect -- it never woke a pending
* on_fetchfile() waiter proxied to that worker, so a mid-wait disconnect
* could only ever be observed at on_fetchfile's own 10s deadline below.
* Called from handle_worker_leave() immediately on disconnect. Unlike
* fetchfile_signal() (keyed by jobid, at most one match), more than one
* fetchfile request can legitimately be in flight against the SAME worker
* at once, so this walks the whole list and signals every match -- a no-op
* if none are pending against `w`. Deliberately takes only g_fetchfile_mu +
* each matched waiter's own mu, never any worker_t lock, so callers do not
* need to hold (or avoid holding) `w`'s lock around this call. */
void fetchfile_signal_worker_disconnect(worker_t* w)
{
fetchfile_pending_t* p;
pthread_mutex_lock(&g_fetchfile_mu);
for (p = g_fetchfile_list; p != NULL; p = p->next) {
if (p->worker != w) continue;
pthread_mutex_lock(&p->mu);
p->status = -ECONNRESET;
p->reply = NULL;
p->done = 1;
pthread_cond_signal(&p->cv);
pthread_mutex_unlock(&p->mu);
}
pthread_mutex_unlock(&g_fetchfile_mu);
}
#ifdef XCE_TESTING
/* Test seam (bug-1693): exercises fetchfile_signal_worker_disconnect()'s
* demux against a real fetchfile_pending_t wait, without needing a full
* on_fetchfile()/real c->jobmgr+c->workermgr fixture (job/worker lookup is
* out of this fix's scope -- only the disconnect-to-waiter signal path is
* under test). Registers a bare pending entry keyed to `w`, blocks with a
* deadline exactly like on_fetchfile's own wait loop below, and reports the
* outcome via *out_status. Returns 1 if woken by a signal, 0 if the
* deadline lapsed first (mirrors pending.done). */
int fetchfile_wait_for_test(worker_t* w, long timeout_ms, int* out_status)
{
fetchfile_pending_t pending = {
.jobid = 0, .worker = w,
.mu = PTHREAD_MUTEX_INITIALIZER, .cv = PTHREAD_COND_INITIALIZER,
.done = 0, .status = -ETIMEDOUT, .reply = NULL, .next = NULL
};
struct timespec deadline;
clock_gettime(CLOCK_REALTIME, &deadline);
deadline.tv_sec += timeout_ms / 1000;
deadline.tv_nsec += (timeout_ms % 1000) * 1000000L;
if (deadline.tv_nsec >= 1000000000L) {
deadline.tv_sec += 1;
deadline.tv_nsec -= 1000000000L;
}
fetchfile_register(&pending);
pthread_mutex_lock(&pending.mu);
while (!pending.done) {
int wrc = pthread_cond_timedwait(&pending.cv, &pending.mu, &deadline);
if (wrc == ETIMEDOUT) break;
}
pthread_mutex_unlock(&pending.mu);
fetchfile_unregister(&pending);
*out_status = pending.status;
return pending.done;
}
#endif
/* ---- on_fetchfile: proxy fetchfile from apiserver to worker ---- */
int on_fetchfile(cluster_t* c, endpoint_t* caller_ep, xce_request_t* request)
@ -3447,6 +3524,12 @@ int on_fetchfile(cluster_t* c, endpoint_t* caller_ep, xce_request_t* request)
return -ENOENT;
}
/* bug-1693: record which worker this request is proxied to (still
* holding w's read lock, so w->ep/identity are stable here) so a later
* disconnect can find and wake this waiter -- see
* fetchfile_signal_worker_disconnect()'s doc comment above. */
pending.worker = w;
/* Register pending before sending to avoid a race with the reply */
fetchfile_register(&pending);
rc = xce_fetchfile_req(w->ep, msg);