lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(libxce): endpoint reference count — kill teardown-vs-dispatch UAF (bug-1298, task 5.8)

Browse source 
The io_worker freed an endpoint (endpoint_killer, cluster.c) on connection
teardown while the vtask dispatcher concurrently read ep->busy
(endpoint_is_task_blocked, netio.c) on a queued vtask — a heap-use-after-free
that ABORTed a controller mid-chaotic-soak (round 4), dropping the cluster
below quorum so no primary could be elected.

Replace the accumulation of ad-hoc ep-lifetime patches (bug-1282 lockless busy,
bug-1197 dispatch_inflight drain + deferred teardown vtask) with one principled
invariant: an endpoint lives until its reference count reaches 0.

  - endpoint_t.refcnt (_Atomic) + heap_alloc flag; endpoint_get starts a heap ep
    at refcnt=1 (the owner ref).
  - endpoint_ref/endpoint_unref: the last unref fini's + frees a heap ep.
    Embedded eps (client ctx->ep, heap_alloc=0) are never freed here — their
    container owns them — so a vtask-balanced ref/unref pair is a safe no-op.
  - vtask_enq takes a per-vtask ref on the bound ep BEFORE publishing the task to
    the dispatcher; run_task / taskq_item_killer release it. So the dispatcher can
    read ep->busy on any queued vtask without the ep being freed underneath it.
  - every HEAP-ep teardown site (endpoint_killer, handle_endpoint_error, the
    vote/prevote connect-fail paths, the vote-ep loser/teardown paths, the standby
    reconnect/eviction paths) now drops its owner ref via endpoint_unref instead of
    raw endpoint_fini+free. Several of those also leaked the ep struct before
    (endpoint_fini-only); unref fixes the leak too.

Validation: controller_unit_test 182 tests / 0 failures (test_ha_vep_teardown
updated — its OFF-path test asserted the old struct leak, now fixed). 6-round
ASAN chaotic soak: ZERO AddressSanitizer reports, ctrl-3 no longer crashes at
round 4, all 6 rounds elect at bounded terms. (The pre-existing iomgr_fini ->
SSL_CTX_free abort under openssl 3.6 in the aggregate test binary is unrelated.)

Note: with the crash gone, the soak now reaches rounds 4-6 for the first time and
surfaces a SEPARATE, pre-existing committed-job-loss/divergence (the bug-795/828
failover catch-up gap; ha-commit-durability domain) — not a refcount regression
(ASAN clean; rounds 1-3 are loss-free).
This commit is contained in:
Rongsong Shen 2026-06-30 14:24:47 +00:00
parent 5be165312a
commit 6e05dc1523
6 changed files with 137 additions and 58 deletions
Download patch file Download diff file Expand all files Collapse all files

19
src/include/netio.h
View file

@ -101,6 +101,18 @@ typedef struct endpoint_t {
time_t ts_last_in;
time_t ts_last_out;
void (*on_connected)(struct endpoint_t *ep); /* fire-once; called in iomgr thread after ssl_handshake_done=1 */
/* bug-1298: reference count governing the lifetime of a HEAP-allocated
* endpoint (endpoint_get). An ep is fini'd + freed exactly when refcnt
* drops to 0. Holders: the creator/owner (endpoint_get sets refcnt=1, the
* teardown path drops it via endpoint_unref) and EVERY ep-bound VTASK_IO
* vtask (vtask_enq +1, run_task / taskq_item_killer -1) — so the vtask
* dispatcher can read ep->busy (endpoint_is_task_blocked) on a queued vtask
* without the io_worker freeing the ep out from under it. Embedded eps
* (struct members like client ctx->ep — heap_alloc==0) participate in the
* count but are NEVER fini'd/freed by endpoint_unref; their owner manages
* their lifetime directly. */
_Atomic int refcnt;
int heap_alloc; /* 1 iff allocated by endpoint_get (unref may free) */
} endpoint_t;
#define INVOKE_MSG_HANDLER(ep) \
@ -155,6 +167,13 @@ extern int endpoint_set_msg_handler(endpoint_t*, msg_handler_t);
extern int endpoint_set_err_handler(endpoint_t*, err_handler_t);
extern int endpoint_set_on_connected(endpoint_t*, void (*cb)(endpoint_t *));
extern void endpoint_fini(endpoint_t*);
/* bug-1298: endpoint refcount. endpoint_ref takes a reference; endpoint_unref
* drops one and, when the last reference is released, fini's the ep and (for
* heap_alloc eps) frees it. Use endpoint_unref in place of endpoint_fini+free
* on every HEAP-ep teardown path so a free never races a vtask still reading
* ep->busy. Safe on NULL. */
extern void endpoint_ref(endpoint_t*);
extern void endpoint_unref(endpoint_t*);
extern int endpoint_set_peer(endpoint_t*, xce_peerinfo_t*);
extern int io_fsm_start(io_fsm_t*, io_packet_t*);