lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(libxce): endpoint reference count — kill teardown-vs-dispatch UAF (bug-1298, task 5.8)

Browse source 
The io_worker freed an endpoint (endpoint_killer, cluster.c) on connection
teardown while the vtask dispatcher concurrently read ep->busy
(endpoint_is_task_blocked, netio.c) on a queued vtask — a heap-use-after-free
that ABORTed a controller mid-chaotic-soak (round 4), dropping the cluster
below quorum so no primary could be elected.

Replace the accumulation of ad-hoc ep-lifetime patches (bug-1282 lockless busy,
bug-1197 dispatch_inflight drain + deferred teardown vtask) with one principled
invariant: an endpoint lives until its reference count reaches 0.

  - endpoint_t.refcnt (_Atomic) + heap_alloc flag; endpoint_get starts a heap ep
    at refcnt=1 (the owner ref).
  - endpoint_ref/endpoint_unref: the last unref fini's + frees a heap ep.
    Embedded eps (client ctx->ep, heap_alloc=0) are never freed here — their
    container owns them — so a vtask-balanced ref/unref pair is a safe no-op.
  - vtask_enq takes a per-vtask ref on the bound ep BEFORE publishing the task to
    the dispatcher; run_task / taskq_item_killer release it. So the dispatcher can
    read ep->busy on any queued vtask without the ep being freed underneath it.
  - every HEAP-ep teardown site (endpoint_killer, handle_endpoint_error, the
    vote/prevote connect-fail paths, the vote-ep loser/teardown paths, the standby
    reconnect/eviction paths) now drops its owner ref via endpoint_unref instead of
    raw endpoint_fini+free. Several of those also leaked the ep struct before
    (endpoint_fini-only); unref fixes the leak too.

Validation: controller_unit_test 182 tests / 0 failures (test_ha_vep_teardown
updated — its OFF-path test asserted the old struct leak, now fixed). 6-round
ASAN chaotic soak: ZERO AddressSanitizer reports, ctrl-3 no longer crashes at
round 4, all 6 rounds elect at bounded terms. (The pre-existing iomgr_fini ->
SSL_CTX_free abort under openssl 3.6 in the aggregate test binary is unrelated.)

Note: with the crash gone, the soak now reaches rounds 4-6 for the first time and
surfaces a SEPARATE, pre-existing committed-job-loss/divergence (the bug-795/828
failover catch-up gap; ha-commit-durability domain) — not a refcount regression
(ASAN clean; rounds 1-3 are loss-free).
This commit is contained in:
Rongsong Shen 2026-06-30 14:24:47 +00:00
parent 5be165312a
commit 6e05dc1523
6 changed files with 137 additions and 58 deletions
Download patch file Download diff file Expand all files Collapse all files

36
src/lib/netio.c
View file

@ -754,9 +754,45 @@ endpoint_t* endpoint_get()
do_endpoint_init(ep);
/* bug-1298: a heap ep starts with the single creator/owner reference. Set
* AFTER do_endpoint_init (which never touches refcnt/heap_alloc, so a later
* endpoint_init() re-init on this same heap ep preserves them). */
ep->heap_alloc = 1;
atomic_init(&ep->refcnt, 1);
return ep;
}
/* bug-1298: take/drop an endpoint reference. The last drop fini's the ep and,
* for a heap ep, frees it. Embedded eps (heap_alloc==0) are never fini'd/freed
* here — their owner manages the struct — so the count is purely advisory for
* them. Both are NULL-safe and lockless (the count is _Atomic). */
void endpoint_ref(endpoint_t* ep)
{
if (ep == NULL) {
return;
}
atomic_fetch_add_explicit(&ep->refcnt, 1, memory_order_relaxed);
}
void endpoint_unref(endpoint_t* ep)
{
if (ep == NULL) {
return;
}
/* acq_rel: the decrement that observes the last reference must happen-after
* every other holder's release so the fini/free sees no concurrent access.
* fini+free ONLY for heap eps: an embedded ep (heap_alloc==0) is owned by its
* container (e.g. client ctx->ep), which fini's it directly — a vtask-balanced
* ref/unref pair on it must NOT fini or free the live struct. */
if (atomic_fetch_sub_explicit(&ep->refcnt, 1, memory_order_acq_rel) == 1 &&
ep->heap_alloc) {
endpoint_fini(ep);
free(ep);
}
}
int endpoint_init(endpoint_t* ep, int domain, int type, int port, int backlog)
{
int fd = -1;

32
src/lib/vtaskmgr.c
View file

@ -68,6 +68,11 @@ void run_task(vtask_t* task)
if (ep != NULL) {
endpoint_unblock_tasks(ep);
/* bug-1298: release the per-vtask reference taken in vtask_enq. If
* this was the last reference (owner already dropped on a torn-down
* ep), endpoint_unref fini's + frees it here — safely, because no
* other vtask references it and the unblock above already completed. */
endpoint_unref(ep);
}
}
@ -187,6 +192,14 @@ static void taskq_item_killer(data_item_t* it)
task->data->u.iodata.pkt = NULL;
}
/* bug-1298: a queued-but-never-executed VTASK_IO still holds the per-vtask
* endpoint reference taken in vtask_enq (run_task never ran to release it).
* Drop it here so the ep's refcount can reach 0 and it is freed. */
if (task->data != NULL && task->data->vtype == VTASK_IO &&
task->data->u.iodata.ep != NULL) {
endpoint_unref(task->data->u.iodata.ep);
}
vtask_fini(task);
free(task);
}
@ -483,13 +496,30 @@ int vtask_enq_classified(vtaskmgr_t* vtaskmgr, vtask_t* task, vtask_class_t cls)
int vtask_enq(vtaskmgr_t* vtaskmgr, vtask_t* task)
{
int rc = 0;
int rc = 0;
endpoint_t* ep = NULL;
/* bug-1298: an ep-bound VTASK_IO holds a reference on its endpoint for the
* whole time it sits in the taskq (the dispatcher reads ep->busy via
* vtask_eligible) and while it runs. Acquire BEFORE dataq_enq publishes the
* task to the dispatcher; release in run_task (executed) or taskq_item_killer
* (dropped at fini). This keeps the io_worker from freeing the ep out from
* under a queued vtask. */
if ((task->data != NULL) && (task->data->vtype == VTASK_IO) &&
(task->data->u.iodata.ep != NULL)) {
ep = task->data->u.iodata.ep;
endpoint_ref(ep);
}
rc = dataq_enq(&(vtaskmgr->taskq), &(task->link));
if (rc >= 0) {
/* wake up task dispatcher */
task->owner = vtaskmgr;
vtaskmgr_wakeup(vtaskmgr);
} else if (ep != NULL) {
/* enqueue failed — the task never entered the queue; drop the ref we
* just took so the caller's own teardown (endpoint_unref) balances. */
endpoint_unref(ep);
}
return rc;