lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693)

Browse source 
Violates this project's own recorded Decision Log entry: "No client
reply-wait timeout as a fix for a hung request -- fail loud at the real
transport/source instead of masking with a deadline backstop."

OPCode_schedAdmin (cluster.c) and on_fetchfile (job.c, the worker proxy
-- cluster.c's own comment says schedAdmin "mirrors on_fetchfile's proxy
pattern", confirming one shared design mistake, not two independent
bugs) both blocked on a multi-second pthread_cond_timedwait, replying
"timeout" on expiry. handle_scheduler_leave()/handle_worker_leave() (the
actual disconnect handlers) only nulled the endpoint pointer -- neither
ever signaled the pending waiter, so a mid-wait disconnect was invisible
until the deadline lapsed. The timeout was the ONLY failure-detection
mechanism, exactly what the decision forbids.

Both sites already had a working signal function on their success path
(schedadmin_signal(), fetchfile_signal()) -- just never wired to the
disconnect handlers. Fixed by calling them from
handle_scheduler_leave()/handle_worker_leave() with a distinguishable
-ECONNRESET outcome, so the waiter wakes immediately on disconnect
instead of only on the deadline. Timeouts kept as defensive backstops
for a genuinely wedged peer that never disconnects cleanly either --
this fix is about not relying SOLELY on the timeout, not removing it.

schedadmin_signal_disconnect() must be called BEFORE handle_scheduler_leave()
acquires c->sched.lock -- the dispatch wait holds that same lock for its
entire duration, so signaling after acquiring it would deadlock.

Found by an /xce-audit-invariants controller pass (Decision Log fidelity
dimension). New test_fetchfile_disconnect.c plus extensions to
test_sched_admin_dispatch.c/test_sched_worker_change.c; RED/GREEN
independently verified against real elapsed-time assertions (pre-fix:
5001ms full-timeout wait; post-fix: <1s immediate wake). Full controller
suite: 1181 tests, 2 failures -- both the pre-existing, unrelated
bug-1561 overlayfs sandbox flake, confirmed via a clean meson-driven run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 14:13:26 +00:00
parent c83b388be2
commit 3eba7c16bf
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
12 changed files with 1032 additions and 424 deletions
Download patch file Download diff file Expand all files Collapse all files

98
src/controller/testsuite/test_sched_worker_change.c
View file

@ -90,6 +90,43 @@ extern int on_scheduler_join(cluster_t*);
* push.c's own stub. */
int check_worker_(xce_worker_t* w) { return (w != NULL) ? 1 : 0; }
/* bug-1693: sched.c's handle_scheduler_leave() now calls
* schedadmin_signal_disconnect() (defined for real in cluster.c, which this
* standalone binary does not link -- see this file's own header comment
* for why real sched.c/dataex.c/egress.c are linked here in isolation).
* check_worker_() just above is a pure link-satisfying no-op stub since
* this file's own tests never reach it at runtime; this one is instead a
* counting SPY -- sched.c IS linked for real here, so handle_scheduler_
* leave() itself (unlike the OPCode_schedAdmin dispatch/wait machinery,
* which lives in cluster.c and is NOT linked here) is exercisable for
* real. test_scheduler_leave_signals_schedadmin_before_clearing_ep below
* uses this spy to prove handle_scheduler_leave() actually calls this
* function (not just that it compiles/links) -- the one piece of the
* bug-1693 fix that test_sched_admin_dispatch.c's own regression test
* (controller_unit_test, where cluster.c IS real but sched.c is stubbed by
* test_config_repl.c) cannot reach, since the two real halves of this fix
* (cluster.c's schedadmin_signal_disconnect() and sched.c's
* handle_scheduler_leave()) are never both linked-for-real in the same
* binary anywhere in this project's current test topology. */
static int g_schedadmin_signal_disconnect_calls;
static cluster_t* g_schedadmin_spy_c; /* set by the test below; NULL elsewhere */
static endpoint_t* g_schedadmin_spy_ep_at_signal_time
= (endpoint_t*)(uintptr_t)0x1; /* poison -- must be overwritten */
int schedadmin_signal_disconnect(void)
{
g_schedadmin_signal_disconnect_calls++;
/* Captures c->sched.ep's value AT THE MOMENT this is called -- proves
* handle_scheduler_leave() calls this BEFORE nulling c->sched.ep, not
* after (see this fix's lock-ordering rationale in sched.c's doc
* comment: calling it after would deadlock against the OPCode_
* schedAdmin dispatch case in production, which holds c->sched.lock for
* its whole wait). */
if (g_schedadmin_spy_c != NULL)
g_schedadmin_spy_ep_at_signal_time = g_schedadmin_spy_c->sched.ep;
return 1;
}
/* ------------------------------------------------------------------ */
/* Fixture */
/* ------------------------------------------------------------------ */
@ -331,6 +368,66 @@ static void test_worker_dead_then_encode_worker_is_asan_clean(void)
"test_worker_dead_then_encode_worker_is_asan_clean: PASS\n");
}
/* ------------------------------------------------------------------ */
/* bug-1693: handle_scheduler_leave() signals schedadmin BEFORE clearing */
/* c->sched.ep (Decision Log fidelity fix) */
/* ------------------------------------------------------------------ */
/* Complements test_sched_admin_dispatch.c's own bug-1693 regression test
* (controller_unit_test): that file proves schedadmin_signal_disconnect()
* (cluster.c, real there) genuinely wakes a real pending OPCode_schedAdmin
* wait immediately, but cannot exercise the real handle_scheduler_leave()
* (sched.c is stubbed by test_config_repl.c in that binary). This binary
* has the opposite shape: sched.c is real, cluster.c is not linked at all,
* so schedadmin_signal_disconnect() above is a counting spy rather than the
* real implementation. Together the two prove both halves of the fix for
* real: the signal function's real effect on a real wait (there), and that
* handle_scheduler_leave() really calls it, unconditionally and before
* clearing c->sched.ep (here). */
static void test_scheduler_leave_signals_schedadmin_before_clearing_ep(void)
{
cluster_t* c = calloc(1, sizeof(cluster_t));
TEST_ASSERT_NOT_NULL(c);
mylock_init(&(c->sched.lock));
endpoint_t sentinel;
memset(&sentinel, 0, sizeof(sentinel));
c->sched.ep = &sentinel;
g_schedadmin_signal_disconnect_calls = 0;
g_schedadmin_spy_c = c;
g_schedadmin_spy_ep_at_signal_time = (endpoint_t*)(uintptr_t)0x1;
int rc = handle_scheduler_leave(c);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, g_schedadmin_signal_disconnect_calls,
"bug-1693 regression: handle_scheduler_leave() must call "
"schedadmin_signal_disconnect() exactly once on every disconnect -- "
"pre-fix, it never called it at all, so a pending OPCode_schedAdmin "
"waiter could only ever be woken by the 5s deadline");
TEST_ASSERT_EQUAL_PTR_MESSAGE(
&sentinel, g_schedadmin_spy_ep_at_signal_time,
"handle_scheduler_leave() must call schedadmin_signal_disconnect() "
"BEFORE nulling c->sched.ep -- calling it after would deadlock "
"against the OPCode_schedAdmin dispatch case in production, which "
"holds c->sched.lock for its entire wait duration (see the "
"lock-ordering rationale in sched.c's handle_scheduler_leave() doc "
"comment)");
TEST_ASSERT_NULL_MESSAGE(c->sched.ep,
"handle_scheduler_leave() must still clear "
"c->sched.ep as before -- this fix only adds "
"the signal call, it does not change the "
"pre-existing ep-clearing behavior");
g_schedadmin_spy_c = NULL;
mylock_fini(&(c->sched.lock));
free(c);
printf(
"test_scheduler_leave_signals_schedadmin_before_clearing_ep: PASS\n");
}
/* ------------------------------------------------------------------ */
/* Standalone Unity main (this binary is its own test executable -- see */
/* meson.build's sched_worker_change_unit_test) */
@ -355,6 +452,7 @@ int main(void)
RUN_TEST(test_worker_dead_leaves_resvals_to_sched_null_and_zero);
RUN_TEST(test_worker_dead_then_encode_worker_is_asan_clean);
RUN_TEST(test_scheduler_leave_signals_schedadmin_before_clearing_ep);
return UNITY_END();
}