fix(controller): fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693)
Violates this project's own recorded Decision Log entry: "No client reply-wait timeout as a fix for a hung request -- fail loud at the real transport/source instead of masking with a deadline backstop." OPCode_schedAdmin (cluster.c) and on_fetchfile (job.c, the worker proxy -- cluster.c's own comment says schedAdmin "mirrors on_fetchfile's proxy pattern", confirming one shared design mistake, not two independent bugs) both blocked on a multi-second pthread_cond_timedwait, replying "timeout" on expiry. handle_scheduler_leave()/handle_worker_leave() (the actual disconnect handlers) only nulled the endpoint pointer -- neither ever signaled the pending waiter, so a mid-wait disconnect was invisible until the deadline lapsed. The timeout was the ONLY failure-detection mechanism, exactly what the decision forbids. Both sites already had a working signal function on their success path (schedadmin_signal(), fetchfile_signal()) -- just never wired to the disconnect handlers. Fixed by calling them from handle_scheduler_leave()/handle_worker_leave() with a distinguishable -ECONNRESET outcome, so the waiter wakes immediately on disconnect instead of only on the deadline. Timeouts kept as defensive backstops for a genuinely wedged peer that never disconnects cleanly either -- this fix is about not relying SOLELY on the timeout, not removing it. schedadmin_signal_disconnect() must be called BEFORE handle_scheduler_leave() acquires c->sched.lock -- the dispatch wait holds that same lock for its entire duration, so signaling after acquiring it would deadlock. Found by an /xce-audit-invariants controller pass (Decision Log fidelity dimension). New test_fetchfile_disconnect.c plus extensions to test_sched_admin_dispatch.c/test_sched_worker_change.c; RED/GREEN independently verified against real elapsed-time assertions (pre-fix: 5001ms full-timeout wait; post-fix: <1s immediate wake). Full controller suite: 1181 tests, 2 failures -- both the pre-existing, unrelated bug-1561 overlayfs sandbox flake, confirmed via a clean meson-driven run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
parent
c83b388be2
commit
3eba7c16bf
12 changed files with 1032 additions and 424 deletions
|
|
@ -667,6 +667,109 @@ static void test_schedadmin_round_trip_business_error(void)
|
|||
printf("test_schedadmin_round_trip_business_error: PASS\n");
|
||||
}
|
||||
|
||||
/* ── bug-1693 regression: scheduler disconnect wakes the waiter immediately
|
||||
* (Decision Log fidelity fix) ───────────────────────────────────────────────
|
||||
*
|
||||
* Before this fix, handle_scheduler_leave() (sched.c) only ever nulled
|
||||
* c->sched.ep on a scheduler disconnect -- it never signalled the pending
|
||||
* OPCode_schedAdmin waiter, so a disconnect mid-wait could only ever be
|
||||
* observed at the FULL SCHEDADMIN_TIMEOUT_MS deadline (5s in production).
|
||||
* That is exactly the Decision Log violation this task fixes: "No client
|
||||
* reply-wait timeout as a fix for a hung request -- fail loud at the real
|
||||
* transport/source instead of masking with a deadline backstop."
|
||||
*
|
||||
* c->sched.ep points at a real (if unconnected) endpoint_t, exactly like
|
||||
* test_schedadmin_round_trip_success above, so the dispatch case's
|
||||
* forward+wait path runs for real. A second thread plays the disconnect by
|
||||
* calling schedadmin_signal_disconnect() (cluster.c) directly -- the exact,
|
||||
* real, unmodified production function handle_scheduler_leave() (sched.c)
|
||||
* calls to wake the waiter, and the entire fix under test here.
|
||||
*
|
||||
* NOT calling handle_scheduler_leave() itself: this file's binary
|
||||
* (controller_unit_test) does not link the real sched.c at all (see
|
||||
* meson.build's comment on sched_push_unit_test/sched_worker_change_unit_
|
||||
* test for why -- sched.c's on_scheduler_join()/collect_confvars() collide
|
||||
* at link time with fake definitions several OTHER controller_unit_test_src
|
||||
* files already provide). test_config_repl.c (also in this binary) already
|
||||
* supplies its OWN no-op handle_scheduler_leave() stub for that reason,
|
||||
* predating this fix -- calling handle_scheduler_leave() here would
|
||||
* silently exercise THAT stub, not sched.c's real implementation, and prove
|
||||
* nothing. handle_scheduler_leave()'s own body is a thin, 2-statement
|
||||
* wrapper (call this function, then null c->sched.ep under c->sched.lock)
|
||||
* -- calling schedadmin_signal_disconnect() directly exercises the entire
|
||||
* real fix mechanism (the new function, and the real dispatch-side wait it
|
||||
* wakes) with zero loss of fidelity versus going through that wrapper. */
|
||||
static void* sa_deliver_scheduler_disconnect(void* argp)
|
||||
{
|
||||
(void)argp;
|
||||
|
||||
/* Same margin as sa_deliver_scheduler_reply above -- comfortably inside
|
||||
* the 5s deadline, giving the main thread time to register the pending
|
||||
* waiter and start blocking on the condvar first. */
|
||||
usleep(200000);
|
||||
|
||||
schedadmin_signal_disconnect();
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static void test_schedadmin_disconnect_wakes_immediately(void)
|
||||
{
|
||||
cluster_t* c = sa_make_cluster();
|
||||
atomic_store(&c->ha.role, CTRL_PRIMARY);
|
||||
|
||||
endpoint_t sched_conn;
|
||||
memset(&sched_conn, 0, sizeof(sched_conn));
|
||||
c->sched.ep = &sched_conn;
|
||||
|
||||
pthread_t tid;
|
||||
TEST_ASSERT_EQUAL_INT(
|
||||
0, pthread_create(&tid, NULL, sa_deliver_scheduler_disconnect, c));
|
||||
|
||||
endpoint_t caller_ep;
|
||||
memset(&caller_ep, 0, sizeof(caller_ep));
|
||||
caller_ep.owner = c;
|
||||
|
||||
io_packet_t* pkt = sa_build_request_pkt();
|
||||
|
||||
uint64_t t0 = sa_mono_ms();
|
||||
int rc = myreq_handler_for_test(&caller_ep, pkt);
|
||||
uint64_t took = sa_mono_ms() - t0;
|
||||
|
||||
pthread_join(tid, NULL);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
0, rc, "the dispatch case always replies, even on a disconnect");
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
took < 1000,
|
||||
"a scheduler disconnect mid-wait must wake the waiter immediately "
|
||||
"(well under the real, unshrunk 5s SCHEDADMIN_TIMEOUT_MS deadline) "
|
||||
"-- this is the regression this test locks: pre-fix, nothing ever "
|
||||
"signalled the waiter on disconnect, so this only ever completed at "
|
||||
"the full 5s deadline");
|
||||
|
||||
xce_request_t decoded;
|
||||
sa_decode_queued_reply(&caller_ep, &decoded);
|
||||
xce_reply_t* rr = decoded.body.replymsg;
|
||||
TEST_ASSERT_NOT_NULL(rr);
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
0, rr->status,
|
||||
"a disconnect is relayed as an ok-RPC SchedAdminReply carrying a "
|
||||
"distinguishable error, not a negative RPC status");
|
||||
TEST_ASSERT_EQUAL_INT(RequestReply_data_schedadminreplymsg, rr->data_which);
|
||||
sched_admin_reply_t* got = rr->data.schedadminreplymsg;
|
||||
TEST_ASSERT_NOT_NULL(got);
|
||||
TEST_ASSERT_EQUAL_INT(0, got->ok);
|
||||
TEST_ASSERT_EQUAL_STRING_MESSAGE(
|
||||
"scheduler disconnected", got->error,
|
||||
"must be distinguishable from \"request timeout\" -- proves this "
|
||||
"woke via the disconnect signal, not the deadline");
|
||||
xce_free_request(&decoded);
|
||||
|
||||
sa_teardown(c);
|
||||
printf("test_schedadmin_disconnect_wakes_immediately: PASS\n");
|
||||
}
|
||||
|
||||
/* ── Suite entry point ──────────────────────────────────────────────────── */
|
||||
void test_sched_admin_dispatch(void)
|
||||
{
|
||||
|
|
@ -678,4 +781,5 @@ void test_sched_admin_dispatch(void)
|
|||
RUN_TEST(test_schedadmin_forward_timeout);
|
||||
RUN_TEST(test_schedadmin_round_trip_rpc_status_only);
|
||||
RUN_TEST(test_schedadmin_round_trip_business_error);
|
||||
RUN_TEST(test_schedadmin_disconnect_wakes_immediately);
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue