The io_worker freed an endpoint (endpoint_killer, cluster.c) on connection
teardown while the vtask dispatcher concurrently read ep->busy
(endpoint_is_task_blocked, netio.c) on a queued vtask — a heap-use-after-free
that ABORTed a controller mid-chaotic-soak (round 4), dropping the cluster
below quorum so no primary could be elected.
Replace the accumulation of ad-hoc ep-lifetime patches (bug-1282 lockless busy,
bug-1197 dispatch_inflight drain + deferred teardown vtask) with one principled
invariant: an endpoint lives until its reference count reaches 0.
- endpoint_t.refcnt (_Atomic) + heap_alloc flag; endpoint_get starts a heap ep
at refcnt=1 (the owner ref).
- endpoint_ref/endpoint_unref: the last unref fini's + frees a heap ep.
Embedded eps (client ctx->ep, heap_alloc=0) are never freed here — their
container owns them — so a vtask-balanced ref/unref pair is a safe no-op.
- vtask_enq takes a per-vtask ref on the bound ep BEFORE publishing the task to
the dispatcher; run_task / taskq_item_killer release it. So the dispatcher can
read ep->busy on any queued vtask without the ep being freed underneath it.
- every HEAP-ep teardown site (endpoint_killer, handle_endpoint_error, the
vote/prevote connect-fail paths, the vote-ep loser/teardown paths, the standby
reconnect/eviction paths) now drops its owner ref via endpoint_unref instead of
raw endpoint_fini+free. Several of those also leaked the ep struct before
(endpoint_fini-only); unref fixes the leak too.
Validation: controller_unit_test 182 tests / 0 failures (test_ha_vep_teardown
updated — its OFF-path test asserted the old struct leak, now fixed). 6-round
ASAN chaotic soak: ZERO AddressSanitizer reports, ctrl-3 no longer crashes at
round 4, all 6 rounds elect at bounded terms. (The pre-existing iomgr_fini ->
SSL_CTX_free abort under openssl 3.6 in the aggregate test binary is unrelated.)
Note: with the crash gone, the soak now reaches rounds 4-6 for the first time and
surfaces a SEPARATE, pre-existing committed-job-loss/divergence (the bug-795/828
failover catch-up gap; ha-commit-durability domain) — not a refcount regression
(ASAN clean; rounds 1-3 are loss-free).
Rongsong Shen
6e05dc1523