lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(libxce): endpoint reference count — kill teardown-vs-dispatch UAF (bug-1298, task 5.8)

Browse source 
The io_worker freed an endpoint (endpoint_killer, cluster.c) on connection
teardown while the vtask dispatcher concurrently read ep->busy
(endpoint_is_task_blocked, netio.c) on a queued vtask — a heap-use-after-free
that ABORTed a controller mid-chaotic-soak (round 4), dropping the cluster
below quorum so no primary could be elected.

Replace the accumulation of ad-hoc ep-lifetime patches (bug-1282 lockless busy,
bug-1197 dispatch_inflight drain + deferred teardown vtask) with one principled
invariant: an endpoint lives until its reference count reaches 0.

  - endpoint_t.refcnt (_Atomic) + heap_alloc flag; endpoint_get starts a heap ep
    at refcnt=1 (the owner ref).
  - endpoint_ref/endpoint_unref: the last unref fini's + frees a heap ep.
    Embedded eps (client ctx->ep, heap_alloc=0) are never freed here — their
    container owns them — so a vtask-balanced ref/unref pair is a safe no-op.
  - vtask_enq takes a per-vtask ref on the bound ep BEFORE publishing the task to
    the dispatcher; run_task / taskq_item_killer release it. So the dispatcher can
    read ep->busy on any queued vtask without the ep being freed underneath it.
  - every HEAP-ep teardown site (endpoint_killer, handle_endpoint_error, the
    vote/prevote connect-fail paths, the vote-ep loser/teardown paths, the standby
    reconnect/eviction paths) now drops its owner ref via endpoint_unref instead of
    raw endpoint_fini+free. Several of those also leaked the ep struct before
    (endpoint_fini-only); unref fixes the leak too.

Validation: controller_unit_test 182 tests / 0 failures (test_ha_vep_teardown
updated — its OFF-path test asserted the old struct leak, now fixed). 6-round
ASAN chaotic soak: ZERO AddressSanitizer reports, ctrl-3 no longer crashes at
round 4, all 6 rounds elect at bounded terms. (The pre-existing iomgr_fini ->
SSL_CTX_free abort under openssl 3.6 in the aggregate test binary is unrelated.)

Note: with the crash gone, the soak now reaches rounds 4-6 for the first time and
surfaces a SEPARATE, pre-existing committed-job-loss/divergence (the bug-795/828
failover catch-up gap; ha-commit-durability domain) — not a refcount regression
(ASAN clean; rounds 1-3 are loss-free).
This commit is contained in:
Rongsong Shen 2026-06-30 14:24:47 +00:00
parent 5be165312a
commit 6e05dc1523
6 changed files with 137 additions and 58 deletions
Download patch file Download diff file Expand all files Collapse all files

20
src/controller/cluster.c
View file

@ -104,13 +104,14 @@ static void handle_endpoint_error(endpoint_t* ep)
}
if (ep->idx >= 0) {
/* zarray_del will finalize ep */
/* zarray_del finalizes ep via endpoint_killer (now endpoint_unref). */
zarray_del(&(c->endpoints), ep->idx);
return;
}
endpoint_fini(ep);
FREEUP(ep);
/* bug-1298: not in c->endpoints — drop the owner ref (deferred fini+free if a
* pool vtask still references it) instead of freeing inline on the io_worker. */
endpoint_unref(ep);
return;
}
@ -757,8 +758,7 @@ static int handle_connection(struct epoll_event* p)
if (priv == NULL) {
PERROR("ep_get_private");
endpoint_fini(ep2);
FREEUP(ep2);
endpoint_unref(ep2); /* bug-1298: refcnt 1->0, fini+free (no queued vtasks yet) */
return -ENOMEM;
}
@ -774,8 +774,7 @@ static int handle_connection(struct epoll_event* p)
PERROR("zarray_store");
FREEUP(priv);
endpoint_fini(ep2);
FREEUP(ep2);
endpoint_unref(ep2); /* bug-1298: refcnt 1->0, fini+free (not yet queued) */
return idx;
}
@ -809,8 +808,11 @@ static void endpoint_killer(void* p)
zarray_clear(&(c->endpoints), ep->idx);
ep->idx = -1;
}
endpoint_fini(ep);
FREEUP(ep);
/* bug-1298: drop the creator/owner reference instead of freeing inline. If
* a pool vtask still references this ep (the io_worker error path runs while
* the dispatcher may be reading ep->busy), the fini+free is deferred to the
* last endpoint_unref — never racing the dispatcher. */
endpoint_unref(ep);
return;
}