fix(controller): fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693)

Violates this project's own recorded Decision Log entry: "No client
reply-wait timeout as a fix for a hung request -- fail loud at the real
transport/source instead of masking with a deadline backstop."

OPCode_schedAdmin (cluster.c) and on_fetchfile (job.c, the worker proxy
-- cluster.c's own comment says schedAdmin "mirrors on_fetchfile's proxy
pattern", confirming one shared design mistake, not two independent
bugs) both blocked on a multi-second pthread_cond_timedwait, replying
"timeout" on expiry. handle_scheduler_leave()/handle_worker_leave() (the
actual disconnect handlers) only nulled the endpoint pointer -- neither
ever signaled the pending waiter, so a mid-wait disconnect was invisible
until the deadline lapsed. The timeout was the ONLY failure-detection
mechanism, exactly what the decision forbids.

Both sites already had a working signal function on their success path
(schedadmin_signal(), fetchfile_signal()) -- just never wired to the
disconnect handlers. Fixed by calling them from
handle_scheduler_leave()/handle_worker_leave() with a distinguishable
-ECONNRESET outcome, so the waiter wakes immediately on disconnect
instead of only on the deadline. Timeouts kept as defensive backstops
for a genuinely wedged peer that never disconnects cleanly either --
this fix is about not relying SOLELY on the timeout, not removing it.

schedadmin_signal_disconnect() must be called BEFORE handle_scheduler_leave()
acquires c->sched.lock -- the dispatch wait holds that same lock for its
entire duration, so signaling after acquiring it would deadlock.

Found by an /xce-audit-invariants controller pass (Decision Log fidelity
dimension). New test_fetchfile_disconnect.c plus extensions to
test_sched_admin_dispatch.c/test_sched_worker_change.c; RED/GREEN
independently verified against real elapsed-time assertions (pre-fix:
5001ms full-timeout wait; post-fix: <1s immediate wake). Full controller
suite: 1181 tests, 2 failures -- both the pre-existing, unrelated
bug-1561 overlayfs sandbox flake, confirmed via a clean meson-driven run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg

src/controller/sched.c

@@ -713,6 +713,27 @@ int handle_scheduler_leave(cluster_t* c)
 {
     /* we need to mark the cluster status */
     TRACE("%s scheduler leave", __func__);
+
+    /* bug-1693 (Decision Log: "No client reply-wait timeout as a fix for a
+     * hung request -- fail loud at the real transport/source instead of
+     * masking with a deadline backstop"). Wake any pending OPCode_schedAdmin
+     * waiter (cluster.c) immediately instead of leaving it to the
+     * SCHEDADMIN_TIMEOUT_MS deadline (5s) -- a no-op if no schedAdmin
+     * request happens to be in flight (schedadmin_signal_disconnect()
+     * returns 0 in that case).
+     *
+     * MUST run BEFORE mylock_lock(&c->sched.lock) below, not after: the
+     * OPCode_schedAdmin dispatch case holds c->sched.lock for its ENTIRE
+     * forward+wait duration (by design -- see that case's own comment in
+     * cluster.c), so if we grabbed c->sched.lock first, this call would
+     * itself block until that handler's own wait resolves one way or the
+     * other -- i.e. nothing would ever be able to signal it early, which is
+     * exactly the bug this fixes. schedadmin_signal_disconnect() only takes
+     * g_schedadmin_mu (cluster.c, file-local) + the pending slot's own mu --
+     * never c->sched.lock -- so calling it here first introduces no new
+     * lock-ordering hazard. */
+    schedadmin_signal_disconnect();
+
     mylock_lock(&(c->sched.lock));
     c->sched.ep = NULL;
     mylock_unlock(&(c->sched.lock));