wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(raft-drive-async): fix bug-1376 cold-start self-deadlock, flip peerctrl+snapshot flags ON

Browse source 
Root cause: peer_ctrl_msg_handler held ep->dispatch_inflight incremented
across the entire peer_ctrl_dispatch_to_pool call, including its inline
fallback (the common cold-start shape when ha->pool isn't wired yet). That
inline body can reach raft_cb_become_follower, which -- now that
cluster_init_phase2 ran synchronously earlier in the same call chain and
wired the pool -- tears down its own loser vote-ep (the very ep whose reply
triggered this call) via the deferred path (vote_ep_teardown_async), whose
endpoint_mark_tearing_down spins waiting for dispatch_inflight==0 on that
SAME ep. That count is held only by this exact call, on this exact thread,
and can't reach 0 until the spin itself returns -- a permanent self-deadlock.

Confirmed live via a gdb thread backtrace: added a GDB_WRAP debug seam to
run-ha-soak.sh (launches a named node directly under `gdb --batch` so it
becomes ptrace-legal as gdb's own direct child under yama scope 1, bypassing
a live-attach restriction), then SIGINT to gdb once the log went stale for a
bounded interrupt + full backtrace. The frame dump showed pool=0x0 at
peer_ctrl_dispatch_to_pool, and the same ep pointer in the nested
vote_ep_teardown_async -> endpoint_mark_tearing_down frame, spinning on
sched_yield().

Fix: moved the dispatch_inflight release into peer_ctrl_dispatch_to_pool
itself, at the exact right point per exit path -- released BEFORE calling
the inline body on every fallback path (pool==NULL, OOM, enqueue failure),
since io_worker is single-threaded and the inline path never enqueues
anything (no pool-worker race left to protect against once inline execution
is committed to); released only AFTER the non-blocking enqueue completes on
the success path, preserving the original protection for the genuinely
concurrent pool-worker case.

New unit test test_vep_self_teardown_does_not_deadlock uses a bounded
pthread_timedjoin_np so a regression fails loud instead of hanging the
suite. RED confirmed via a temporary manual revert (ETIMEDOUT) before
restoring the fix.

Live-verified with both flags ON: --force-snapshot soak (the exact repro
shape) completes without hanging across repeated runs, confirms real
snapshot installs, zero divergence, zero job loss; a 6-round --chaotic soak
also passed clean; full controller suite green throughout.

g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch_enabled
now default ON in production, closing raft-drive-async tasks 3.1 and 3.2.
g_drive_async_configpush_offload stays OFF -- a distinct, still-open hazard
in push_config_transport_only's endpoint handling (task 3.3, unrelated to
this bug).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014dZVSx3CdBjys9BB6hqpxP
This commit is contained in:
Rongsong Shen 2026-07-02 07:18:42 +00:00
parent 10f92a5d49
commit 1ce007814d
5 changed files with 272 additions and 70 deletions
Download patch file Download diff file Expand all files Collapse all files

86
src/controller/testsuite/test_ha_vep_teardown.c
View file

@ -538,6 +538,91 @@ static void test_vep_mark_drains_inflight_dispatch(void)
printf("test_vep_mark_drains_inflight_dispatch: PASS\n");
}
/* ── Test (f): bug-1376 SELF-TEARDOWN — must not self-deadlock ─────────────
*
* A node's OWN in-flight peer-control dispatch can, via its routing body,
* decide to tear down the VERY SAME ep the dispatch is currently processing
* (the live signature: a node's own vote-request ep receives the reply that
* makes it become follower, and the loser loop closes that same ep). Before
* the bug-1376 fix, peer_ctrl_msg_handler held ep->dispatch_inflight across
* the ENTIRE inline body call; vote_ep_teardown_async's endpoint_mark_
* tearing_down then spun forever waiting for that same count — held by this
* same thread, unable to decrement until the spin itself returns — a
* permanent self-deadlock. Reproduced with a bounded-timeout join: the
* dispatch runs on a helper thread; the test FAILS (rather than hanging the
* whole suite) if it does not return within the timeout.
*/
static void *self_teardown_dispatch_thread(void *p)
{
dispatch_arg_t *a = (dispatch_arg_t *)p;
io_packet_t *pkt = repl_catchup_done_pkt_for_test();
peer_ctrl_msg_handler_for_test(a->ep, pkt);
atomic_store(&a->done, 1);
return NULL;
}
static ha_state_t *g_self_teardown_ha = NULL;
static endpoint_t *g_self_teardown_ep = NULL;
/* Fires at the top of the inline routing body (before role dispatch) — tears
* down the SAME ep this dispatch is currently processing, the self-referential
* case the fix must resolve. */
static void self_teardown_body_probe(endpoint_t *ep)
{
(void)ep;
ha_vote_ep_teardown_for_test(g_self_teardown_ha, g_self_teardown_ep);
}
static void test_vep_self_teardown_does_not_deadlock(void)
{
reset_oracles();
ha_state_t ha;
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &g_cfg, 1));
/* CTRL_INIT → the inline body hits the default case (drop + free pkt)
* after the probe runs, so the stand-in vote_ep_ctx_t is sufficient. */
atomic_store_explicit(&ha.role, (int32_t)CTRL_INIT, memory_order_release);
ha.pool = NULL; /* matches the live cold-start repro (bug-1376) */
endpoint_t *ep = endpoint_get();
TEST_ASSERT_NOT_NULL(ep);
ep->iomgr = NULL;
ep->fd = -1;
struct mini_vctx *v = calloc(1, sizeof(*v));
TEST_ASSERT_NOT_NULL(v);
v->ha = &ha;
ep->data = v;
ha_set_peerctrl_dispatch_for_test(1);
g_self_teardown_ha = &ha;
g_self_teardown_ep = ep;
ha_peerctrl_body_probe = self_teardown_body_probe;
dispatch_arg_t darg = { .ep = ep };
atomic_store(&darg.done, 0);
pthread_t t;
TEST_ASSERT_EQUAL_INT(0,
pthread_create(&t, NULL, self_teardown_dispatch_thread, &darg));
struct timespec deadline;
clock_gettime(CLOCK_REALTIME, &deadline);
deadline.tv_sec += 5;
int jrc = pthread_timedjoin_np(t, NULL, &deadline);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, jrc,
"bug-1376: self-teardown (an ep's own dispatch tearing down the same "
"ep) must not deadlock — dispatch_inflight must be released before "
"the inline body runs, not held across it");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, atomic_load(&darg.done),
"the dispatch thread must have completed");
ha_peerctrl_body_probe = NULL;
ha_set_peerctrl_dispatch_for_test(0);
g_self_teardown_ha = NULL;
g_self_teardown_ep = NULL;
ha_fini(&ha);
printf("test_vep_self_teardown_does_not_deadlock: PASS\n");
}
/* ── Entry point ─────────────────────────────────────────────────────────── */
void test_ha_vep_teardown(void)
@ -547,4 +632,5 @@ void test_ha_vep_teardown(void)
RUN_TEST(test_vep_teardown_vtask_disarms_unblock);
RUN_TEST(test_vep_off_path_uses_plain_fini);
RUN_TEST(test_vep_mark_drains_inflight_dispatch);
RUN_TEST(test_vep_self_teardown_does_not_deadlock);
}