lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693)

Browse source 
Violates this project's own recorded Decision Log entry: "No client
reply-wait timeout as a fix for a hung request -- fail loud at the real
transport/source instead of masking with a deadline backstop."

OPCode_schedAdmin (cluster.c) and on_fetchfile (job.c, the worker proxy
-- cluster.c's own comment says schedAdmin "mirrors on_fetchfile's proxy
pattern", confirming one shared design mistake, not two independent
bugs) both blocked on a multi-second pthread_cond_timedwait, replying
"timeout" on expiry. handle_scheduler_leave()/handle_worker_leave() (the
actual disconnect handlers) only nulled the endpoint pointer -- neither
ever signaled the pending waiter, so a mid-wait disconnect was invisible
until the deadline lapsed. The timeout was the ONLY failure-detection
mechanism, exactly what the decision forbids.

Both sites already had a working signal function on their success path
(schedadmin_signal(), fetchfile_signal()) -- just never wired to the
disconnect handlers. Fixed by calling them from
handle_scheduler_leave()/handle_worker_leave() with a distinguishable
-ECONNRESET outcome, so the waiter wakes immediately on disconnect
instead of only on the deadline. Timeouts kept as defensive backstops
for a genuinely wedged peer that never disconnects cleanly either --
this fix is about not relying SOLELY on the timeout, not removing it.

schedadmin_signal_disconnect() must be called BEFORE handle_scheduler_leave()
acquires c->sched.lock -- the dispatch wait holds that same lock for its
entire duration, so signaling after acquiring it would deadlock.

Found by an /xce-audit-invariants controller pass (Decision Log fidelity
dimension). New test_fetchfile_disconnect.c plus extensions to
test_sched_admin_dispatch.c/test_sched_worker_change.c; RED/GREEN
independently verified against real elapsed-time assertions (pre-fix:
5001ms full-timeout wait; post-fix: <1s immediate wake). Full controller
suite: 1181 tests, 2 failures -- both the pre-existing, unrelated
bug-1561 overlayfs sandbox flake, confirmed via a clean meson-driven run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 14:13:26 +00:00
parent c83b388be2
commit 3eba7c16bf
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
12 changed files with 1032 additions and 424 deletions
Download patch file Download diff file Expand all files Collapse all files

17
src/controller/worker.c
View file

@ -775,6 +775,23 @@ int handle_worker_leave(cluster_t* c, int idx)
unlock(w);
/* bug-1693 (Decision Log: "No client reply-wait timeout as a fix for a
* hung request -- fail loud at the real transport/source instead of
* masking with a deadline backstop"). Wake any on_fetchfile() waiter
* (job.c) proxied to this worker immediately instead of leaving it to
* on_fetchfile's own 10s deadline -- mirrors handle_scheduler_leave's
* identical schedadmin_signal_disconnect() fix in sched.c (same
* Decision Log rule, same shared design mistake -- cluster.c's
* OPCode_schedAdmin case comment already notes it "mirrors
* on_fetchfile's proxy pattern"). Deliberately called AFTER releasing
* w's lock: fetchfile_signal_worker_disconnect() only takes
* g_fetchfile_mu (job.c, file-local) + each matched waiter's own mu,
* never any worker_t lock, so there is no ordering dependency, and `w`
* remains a valid pointer for the comparison (this function's vget'd
* reference is still held). A no-op if no fetchfile request happens to
* be pending against this worker. */
fetchfile_signal_worker_disconnect(w);
return 0;
}