fix(controller): fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693)

Violates this project's own recorded Decision Log entry: "No client
reply-wait timeout as a fix for a hung request -- fail loud at the real
transport/source instead of masking with a deadline backstop."

OPCode_schedAdmin (cluster.c) and on_fetchfile (job.c, the worker proxy
-- cluster.c's own comment says schedAdmin "mirrors on_fetchfile's proxy
pattern", confirming one shared design mistake, not two independent
bugs) both blocked on a multi-second pthread_cond_timedwait, replying
"timeout" on expiry. handle_scheduler_leave()/handle_worker_leave() (the
actual disconnect handlers) only nulled the endpoint pointer -- neither
ever signaled the pending waiter, so a mid-wait disconnect was invisible
until the deadline lapsed. The timeout was the ONLY failure-detection
mechanism, exactly what the decision forbids.

Both sites already had a working signal function on their success path
(schedadmin_signal(), fetchfile_signal()) -- just never wired to the
disconnect handlers. Fixed by calling them from
handle_scheduler_leave()/handle_worker_leave() with a distinguishable
-ECONNRESET outcome, so the waiter wakes immediately on disconnect
instead of only on the deadline. Timeouts kept as defensive backstops
for a genuinely wedged peer that never disconnects cleanly either --
this fix is about not relying SOLELY on the timeout, not removing it.

schedadmin_signal_disconnect() must be called BEFORE handle_scheduler_leave()
acquires c->sched.lock -- the dispatch wait holds that same lock for its
entire duration, so signaling after acquiring it would deadlock.

Found by an /xce-audit-invariants controller pass (Decision Log fidelity
dimension). New test_fetchfile_disconnect.c plus extensions to
test_sched_admin_dispatch.c/test_sched_worker_change.c; RED/GREEN
independently verified against real elapsed-time assertions (pre-fix:
5001ms full-timeout wait; post-fix: <1s immediate wake). Full controller
suite: 1181 tests, 2 failures -- both the pre-existing, unrelated
bug-1561 overlayfs sandbox flake, confirmed via a clean meson-driven run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 14:13:26 +00:00
parent c83b388be2
commit 3eba7c16bf
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
12 changed files with 1032 additions and 424 deletions

View file

@ -64,6 +64,15 @@ extern int on_scheduler_join(cluster_t*);
* rather than linking all of opget.c, mirroring opget.c's own trivial body. */
int check_worker_(xce_worker_t* w) { return (w != NULL) ? 1 : 0; }
/* bug-1693: sched.c's handle_scheduler_leave() now calls
* schedadmin_signal_disconnect() (defined in cluster.c, which this
* standalone binary does not link -- see this file's own comment above for
* why). This test never triggers a scheduler-leave/disconnect path, so the
* real behavior is never exercised here; the symbol only needs to resolve
* at link time, mirroring check_worker_()'s identical stub rationale just
* above. */
int schedadmin_signal_disconnect(void) { return 0; }
/* ------------------------------------------------------------------ */
/* Fixture */
/* ------------------------------------------------------------------ */