fix(controller): fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693)

Violates this project's own recorded Decision Log entry: "No client
reply-wait timeout as a fix for a hung request -- fail loud at the real
transport/source instead of masking with a deadline backstop."

OPCode_schedAdmin (cluster.c) and on_fetchfile (job.c, the worker proxy
-- cluster.c's own comment says schedAdmin "mirrors on_fetchfile's proxy
pattern", confirming one shared design mistake, not two independent
bugs) both blocked on a multi-second pthread_cond_timedwait, replying
"timeout" on expiry. handle_scheduler_leave()/handle_worker_leave() (the
actual disconnect handlers) only nulled the endpoint pointer -- neither
ever signaled the pending waiter, so a mid-wait disconnect was invisible
until the deadline lapsed. The timeout was the ONLY failure-detection
mechanism, exactly what the decision forbids.

Both sites already had a working signal function on their success path
(schedadmin_signal(), fetchfile_signal()) -- just never wired to the
disconnect handlers. Fixed by calling them from
handle_scheduler_leave()/handle_worker_leave() with a distinguishable
-ECONNRESET outcome, so the waiter wakes immediately on disconnect
instead of only on the deadline. Timeouts kept as defensive backstops
for a genuinely wedged peer that never disconnects cleanly either --
this fix is about not relying SOLELY on the timeout, not removing it.

schedadmin_signal_disconnect() must be called BEFORE handle_scheduler_leave()
acquires c->sched.lock -- the dispatch wait holds that same lock for its
entire duration, so signaling after acquiring it would deadlock.

Found by an /xce-audit-invariants controller pass (Decision Log fidelity
dimension). New test_fetchfile_disconnect.c plus extensions to
test_sched_admin_dispatch.c/test_sched_worker_change.c; RED/GREEN
independently verified against real elapsed-time assertions (pre-fix:
5001ms full-timeout wait; post-fix: <1s immediate wake). Full controller
suite: 1181 tests, 2 failures -- both the pre-existing, unrelated
bug-1561 overlayfs sandbox flake, confirmed via a clean meson-driven run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg

src/controller/cluster.c

@@ -285,12 +285,13 @@ static void schedadmin_unregister(schedadmin_pending_t* p)
 
 /* Signal the pending schedAdmin waiter (if any) with the scheduler's reply.
  * Called from the OPCode_reply case below when a SchedAdminReply arrives on
- * c->sched.ep. Returns 1 if a waiter was found and signaled (ownership of
- * `sar` transfers to it -- the caller must NULL its own copy so the normal
- * xce_free_request path does not also free it), 0 if no waiter was
- * registered (e.g. a late reply arriving after this controller already gave
- * up and replied "request timeout" to xceadmin -- `sar` stays owned by the
- * caller in that case, freed normally). */
+ * c->sched.ep, AND (bug-1693) from schedadmin_signal_disconnect() below when
+ * the scheduler disconnects mid-wait. Returns 1 if a waiter was found and
+ * signaled (ownership of `sar` transfers to it -- the caller must NULL its
+ * own copy so the normal xce_free_request path does not also free it), 0 if
+ * no waiter was registered (e.g. a late reply arriving after this
+ * controller already gave up and replied "request timeout" to xceadmin --
+ * `sar` stays owned by the caller in that case, freed normally). */
 static int schedadmin_signal(int status, sched_admin_reply_t* sar)
 {
     pthread_mutex_lock(&g_schedadmin_mu);
@@ -309,6 +310,34 @@ static int schedadmin_signal(int status, sched_admin_reply_t* sar)
     return 1;
 }
 
+/* bug-1693 (Decision Log: "No client reply-wait timeout as a fix for a hung
+ * request -- fail loud at the real transport/source instead of masking with
+ * a deadline backstop"). Before this fix, handle_scheduler_leave() (sched.c)
+ * only ever nulled c->sched.ep on a scheduler disconnect -- it never woke a
+ * pending OPCode_schedAdmin waiter, so a mid-wait disconnect could only ever
+ * be observed at the full SCHEDADMIN_TIMEOUT_MS deadline (5s), exactly the
+ * "sole failure-detection mechanism is a timeout" shape that Decision Log
+ * entry forbids. This is the narrow, non-static entry point
+ * handle_scheduler_leave() calls to fix that -- a thin wrapper around
+ * schedadmin_signal() with a fixed, distinguishable -ECONNRESET status and
+ * no reply payload (there is no real SchedAdminReply on a disconnect).
+ * Deliberately does NOT expose schedadmin_signal() itself (or the
+ * sched_admin_reply_t type it takes) outside this file -- cluster.h has no
+ * other need for typemap.h, and sched.c only ever needs to signal the
+ * disconnect outcome, never an arbitrary reply.
+ *
+ * Lock-ordering note (see handle_scheduler_leave()'s own doc comment in
+ * sched.c for the full argument): schedadmin_signal() takes only
+ * g_schedadmin_mu + the pending slot's own mu -- never c->sched.lock. The
+ * OPCode_schedAdmin case above holds c->sched.lock for its ENTIRE
+ * forward+wait duration, so calling this function requires no lock
+ * c->sched.lock's holder could be blocked on, and introduces no new
+ * lock-ordering hazard. */
+int schedadmin_signal_disconnect(void)
+{
+    return schedadmin_signal(-ECONNRESET, NULL);
+}
+
 #ifdef XCE_TESTING
 /*
  * schedadmin_set_timeout_ms_for_test / SCHEDADMIN_TIMEOUT_MS (task 10.2, Req
@@ -997,6 +1026,22 @@ static int myreq_handler(vtask_t* task)
             reply.data.schedadminreplymsg = got_reply;
             xce_reply(ep, c->myhost, &reply);
             free_SchedAdminReply_ptr(&got_reply);
+        } else if (got_done && got_status == -ECONNRESET) {
+            /* bug-1693: woken by schedadmin_signal_disconnect()
+             * (handle_scheduler_leave(), sched.c) rather than a genuine
+             * scheduler reply -- the scheduler disconnected mid-wait. Relay
+             * a structured SchedAdminReply, the same shape as the "not
+             * connected"/"request timeout" branches above and below, with a
+             * distinguishable error string so this outcome is never
+             * conflated with either in logs or tests. */
+            sched_admin_reply_t disc_reply;
+            memset(&disc_reply, 0, sizeof(disc_reply));
+            disc_reply.error = "scheduler disconnected";
+
+            reply.status     = 0;
+            reply.data_which = RequestReply_data_schedadminreplymsg;
+            reply.data.schedadminreplymsg = &disc_reply;
+            xce_reply(ep, c->myhost, &reply);
         } else if (got_done) {
             /* Scheduler replied at the RPC level with no SchedAdminReply
              * payload (unexpected, but handled loudly, never dropped) --