lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): standby-endpoint err handler freed ep->data under a live vtask (bug-1839)

Browse source 
Live 3-node HA soak crashed xce_controller: SIGSEGV in
peer_ctrl_msg_handler_inline (ha.c:7035) dereferencing a NULL
ep->data. gdb analysis of the actual core dump showed the endpoint
itself fully alive (refcnt=191964, tearing_down=0) while ep->data had
been synchronously freed and nulled by standby_ep_err_handler_primary,
the endpoint's err_handler, with zero coordination with the vtask pool
that was concurrently dispatching a peer-control message for the same
endpoint (g_drive_async_peerctrl_dispatch_enabled defaults ON).

The sibling handlers (vote_ep_err_handler, pre_vote_ep_err_handler)
already avoid this exact hazard, with an explicit comment: "Do NOT
free vctx here -- peer_ctrl_msg_handler may still reference ep->data."
standby_ep_err_handler_primary was never updated to the same
discipline when the vtask-pool dispatch machinery (bug-1197/bug-1298)
was introduced.

Fix: route standby_ep_err_handler_primary through the same
already-tested vote_ep_teardown_async() deferred, per-endpoint-FIFO-
serialized teardown that vote-endpoint demotion already uses, plus an
idempotency guard (endpoint_is_tearing_down) since the old "ep->data
== NULL" reentry check no longer applies once the free is deferred.
Also fixes a second-order double-unref this surfaced in
on_standby_connected's stale-duplicate eviction path, caught by ASAN.

TDD: two new tests prove the teardown is deferred until an in-flight
same-ep vtask completes, and that a repeated error callback on the
same endpoint is a safe no-op. Full controller suite: 1219/1219, zero
regressions.

Also logs bug-1840 (a separate scheduler heap-corruption crash from
the same soak run) as not-fixed pending an ASAN reproduction -- a
non-instrumented core dump cannot localize glibc's malloc_consolidate
abort back to the actual out-of-bounds write.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-22 03:21:14 +00:00
parent 4f94638a1d
commit e1da9f1e9b
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
7 changed files with 299 additions and 38 deletions
Download patch file Download diff file Expand all files Collapse all files

38
src/controller/ha.c
View file

@ -9462,14 +9462,14 @@ static void on_standby_connected(endpoint_t *ep)
WARN("HA: peer[%d] reconnected — evicting stale duplicate outbound "
"channel to keep exactly one live connection (Req 2.6)",
sctx->peer_idx);
if (stale_same_peer->iomgr)
iomgr_unmonitor((iomgr_t *)stale_same_peer->iomgr, stale_same_peer);
/* bug-1839: standby_ep_err_handler_primary now owns the FULL
* teardown itself (mark tearing_down, iomgr_unmonitor if needed,
* defer the owner-ref drop via vote_ep_teardown_async so a same-ep
* peer-control vtask already queued/in-flight is not read after
* free) — it used to only free ep->data/null it, leaving the caller
* to iomgr_unmonitor + endpoint_unref manually. Do NOT also
* unmonitor/unref here: doing so double-drops the owner reference. */
(void)standby_ep_err_handler_primary(stale_same_peer);
/* err-handler set ep->data = NULL and freed sctx; bug-1298: drop the
* owner ref — endpoint_unref reclaims outq/ssl/packets and frees the
* calloc'd struct when the last reference goes (deferred if a pool vtask
* still references this just-unmonitored ep). */
endpoint_unref(stale_same_peer);
#ifdef XCE_TESTING
atomic_fetch_add(&ha_standby_evicted_count_for_test, 1);
#endif
@ -9542,7 +9542,18 @@ static void on_standby_connected(endpoint_t *ep)
*/
static int standby_ep_err_handler_primary(void *ep_raw)
{
endpoint_t *ep = (endpoint_t *)ep_raw;
endpoint_t *ep = (endpoint_t *)ep_raw;
/* bug-1839: idempotency guard. Once this handler stops nulling ep->data
* synchronously (below), a second handle_error() callback for the same
* ep (e.g. a read error and a write error both observed before
* iomgr_unmonitor takes effect) must not re-enter vote_ep_teardown_async
* and double-drop the endpoint's owner reference. endpoint_mark_tearing_
* down (inside vote_ep_teardown_async) sets this flag synchronously
* before returning, so a repeated call deterministically observes it. */
if (endpoint_is_tearing_down(ep))
return 0;
standby_ep_ctx_t *sctx = (standby_ep_ctx_t *)ep->data;
if (!sctx) return 0;
@ -9566,8 +9577,15 @@ static int standby_ep_err_handler_primary(void *ep_raw)
if (c)
repl_remove_standby(&c->persist.repl, ep);
free(sctx);
ep->data = NULL;
/* bug-1839: do not free sctx / null ep->data synchronously here — a
* peer-control vtask for this same ep may already be queued or in
* flight (peer_ctrl_msg_handler_inline reads ep->data), same hazard
* vote_ep_err_handler's own comment documents for vote_ep_ctx_t just
* above. Defer to the same safe, per-ep-FIFO-serialized teardown
* vote_ep_teardown_async already uses for promoted vote-endpoint
* demotion (raft_cb_become_follower) — endpoint_fini's own
* FREEUP(ep->data) frees sctx once the last reference releases. */
vote_ep_teardown_async(ha, ep);
return 0;
}