fix(libxce): endpoint reference count — kill teardown-vs-dispatch UAF (bug-1298, task 5.8)

The io_worker freed an endpoint (endpoint_killer, cluster.c) on connection
teardown while the vtask dispatcher concurrently read ep->busy
(endpoint_is_task_blocked, netio.c) on a queued vtask — a heap-use-after-free
that ABORTed a controller mid-chaotic-soak (round 4), dropping the cluster
below quorum so no primary could be elected.

Replace the accumulation of ad-hoc ep-lifetime patches (bug-1282 lockless busy,
bug-1197 dispatch_inflight drain + deferred teardown vtask) with one principled
invariant: an endpoint lives until its reference count reaches 0.

  - endpoint_t.refcnt (_Atomic) + heap_alloc flag; endpoint_get starts a heap ep
    at refcnt=1 (the owner ref).
  - endpoint_ref/endpoint_unref: the last unref fini's + frees a heap ep.
    Embedded eps (client ctx->ep, heap_alloc=0) are never freed here — their
    container owns them — so a vtask-balanced ref/unref pair is a safe no-op.
  - vtask_enq takes a per-vtask ref on the bound ep BEFORE publishing the task to
    the dispatcher; run_task / taskq_item_killer release it. So the dispatcher can
    read ep->busy on any queued vtask without the ep being freed underneath it.
  - every HEAP-ep teardown site (endpoint_killer, handle_endpoint_error, the
    vote/prevote connect-fail paths, the vote-ep loser/teardown paths, the standby
    reconnect/eviction paths) now drops its owner ref via endpoint_unref instead of
    raw endpoint_fini+free. Several of those also leaked the ep struct before
    (endpoint_fini-only); unref fixes the leak too.

Validation: controller_unit_test 182 tests / 0 failures (test_ha_vep_teardown
updated — its OFF-path test asserted the old struct leak, now fixed). 6-round
ASAN chaotic soak: ZERO AddressSanitizer reports, ctrl-3 no longer crashes at
round 4, all 6 rounds elect at bounded terms. (The pre-existing iomgr_fini ->
SSL_CTX_free abort under openssl 3.6 in the aggregate test binary is unrelated.)

Note: with the crash gone, the soak now reaches rounds 4-6 for the first time and
surfaces a SEPARATE, pre-existing committed-job-loss/divergence (the bug-795/828
failover catch-up gap; ha-commit-durability domain) — not a refcount regression
(ASAN clean; rounds 1-3 are loss-free).
This commit is contained in:
Rongsong Shen 2026-06-30 14:24:47 +00:00
parent 5be165312a
commit 6e05dc1523
6 changed files with 137 additions and 58 deletions

View file

@ -320,10 +320,10 @@ static void test_vep_late_event_dropped(void)
ha_peerctrl_body_probe = NULL;
ha_set_peerctrl_dispatch_for_test(0);
/* ep was not torn down by a vtask here — free it ourselves (pkt was freed
* by the drop path). */
endpoint_fini(ep);
free(ep);
/* ep was not torn down by a vtask here (late event dropped, refcnt==1) —
* drop the owner ref ourselves; endpoint_unref fini's + frees it (pkt was
* freed by the drop path). bug-1298: use unref, not raw fini+free. */
endpoint_unref(ep);
vtaskmgr_fini(&pool);
iomgr_fini(&iomgr);
@ -402,11 +402,12 @@ static void test_vep_off_path_uses_plain_fini(void)
TEST_ASSERT_EQUAL_INT_MESSAGE(0, atomic_load(&g_teardown_done),
"OFF path must NOT run the async teardown vtask");
TEST_ASSERT_NULL_MESSAGE(ectx->votes[0].ep,
"loser loop must clear votes[i].ep after endpoint_fini");
"loser loop must clear votes[i].ep after endpoint_unref");
/* OFF path called endpoint_fini(ep) (freed ep->data) but NOT free(ep) — the
* legacy per-lost-election leak. Free the struct here so the test is clean. */
free(ep);
/* bug-1298: the OFF path now drops the owner ref via endpoint_unref — with no
* pool vtask referencing the ep (refcnt==1) that fini's AND frees it here,
* fixing the legacy per-lost-election struct leak. So the test must NOT
* free(ep) again (that was the old leak-cleanup; it is now a double-free). */
atomic_store_explicit(&ha.elect_ctx, NULL, memory_order_release);
free(ectx);