myxce/.wolf/bugs/bug-1447.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	legacy	gitea_issue	gitea_synced_status
bug-1447	fixed	2026-09-24		lib	crash	2026-07-18T01:05:02Z	2026-09-19T09:00:00Z	1	src/lib/netio.c (handle_ssl_handshake / io_handler / io_worker -- crash site shared iomgr epoll dispatch); src/controller/ha.c (ha_connect_prep_peer/ha_connect_finish_peer/reconnect_lane_fn raft-drive-async Task 2.1/2.2 reconnect-offload -- suspected trigger); src/controller/cluster.c (handle_connection / endpoint_accept -- inbound accept path alternate suspect)	ASAN FIXED SEGV endpoint_ref handle_ssl_handshake io_worker kiro-validate-impl netio.c on_standby_connected openssl raft-drive-async reconnect-offload regression-test round-9 same-batch-uaf sequential-soak ssl3_read_bytes	bug-1282 bug-1286 bug-1298	true	32	fixed

bug-1447: /kiro-validate-impl raft-drive-async: 10-round SEQUENTIAL HA soak (the fully-supported…

Description

/kiro-validate-impl raft-drive-async: 10-round SEQUENTIAL HA soak (the fully-supported single-failover lifecycle, hard-fail-on-any-loss) failed at round 9/10 -- 'FAIL: round 9: post-failover submit did not get 3 acks' (0/3 acked across 5 retry attempts). Root traced to ctrl-3 (a standby) crashing: AddressSanitizer DEADLYSIGNAL SEGV on address 0x40 (near-null, READ) in openssl's ssl3_read_bytes <- tls_get_message_header <- state_machine <- handle_ssl_handshake (src/lib/netio.c:548) <- io_handler (netio.c:588) <- io_worker (netio.c:633), thread T2 (an io_worker thread). With ctrl-1 already killed (the round's own kill target) and ctrl-3 now also dead, no 2-of-3 quorum remained for ctrl-2 (freshly elected PRIMARY, term=11) to commit writes -- explaining the observed 0/3-acks symptom as a downstream consequence, not the primary defect.

Environment

Not recorded (migrated from buglog.json).

Reproduction

Not recorded (migrated from buglog.json).

Root cause

CONFIRMED (deterministic regression test + live soak, not speculative). The earlier 'unsynchronized concurrent SSL_do_handshake from two io_worker threads' hypothesis is WRONG and is retracted: iomgr_start() creates exactly ONE io_worker thread per iomgr_t (src/lib/netio.c:iomgr_start), and a single controller process has exactly one such iomgr/thread for all peer connections -- there is no second io_worker thread to race against.

REAL mechanism: io_worker's epoll dispatch loop processes every ready fd from ONE epoll_wait() batch in a plain for-loop calling io_handler() per event (src/lib/netio.c, old io_worker ~line 612-640). A handler for an EARLIER event in that SAME batch can free a DIFFERENT endpoint that also has a ready event LATER in the SAME batch. In production this is exactly src/controller/ha.c:8257-8331 on_standby_connected(), the handshake-complete callback for a freshly-reconnected outbound standby channel (raft-drive-async Task 2.1/2.2's reconnect-offload path): when it detects a stale duplicate outbound channel to the same peer (ha.c:8286-8303, 'Different ep, same peer -> stale duplicate to evict'), it calls iomgr_unmonitor() (ha.c:8320-8321) then endpoint_unref() (ha.c:8327) on the STALE endpoint. iomgr_unmonitor()'s epoll_ctl(EPOLL_CTL_DEL) only stops a FUTURE epoll_wait() from returning events for that fd -- it does NOT invalidate an event for that SAME stale endpoint that is already sitting in the CURRENT batch's stack-local events[] array (very plausible during a reconnect burst, where several peer connections' fds become ready close together right after a new primary election). endpoint_unref() reaching refcnt 0 immediately SSL_free()s ep->ssl and free()s the endpoint_t. The loop's next iteration then calls io_handler() for that now-freed endpoint -- a heap-use-after-free that reads freed memory as 'ep', and (if it still looks like a plausible pointer chain) hands a freed/reused ep->ssl to SSL_do_handshake(), producing exactly the observed ASAN DEADLYSIGNAL SEGV deep inside OpenSSL's ssl3_read_bytes rather than an immediate top-of-function null-ssl crash.

This is a genuine gap in the bug-1282/1286/1298 hardening family: INVOKE_MSG_HANDLER (netio.h) already refs/unrefs an endpoint around its OWN msg_handler call (bug-1298 note in that macro's comment) so a handler cannot free the SAME ep it is currently dispatched on -- but that bracket never covered a DIFFERENT, already-batched endpoint torn down as a side effect (on_standby_connected's cross-endpoint eviction). io_worker's own epoll dispatch loop held no reference of any kind on the endpoints in its batch. Reproduced deterministically (no live soak needed to hit it) via a new unit test that drives io_worker's real batch-dispatch loop with a synthetic 2-event epoll batch where event 0's handler endpoint_unref()s event 1's endpoint -- confirmed RED pre-fix: ASAN heap-use-after-free at src/lib/netio.c:587 (io_handler reading ep->ssl_handshake_done), freed by src/lib/netio.c:843 (endpoint_unref) called from the injected handler, exactly mirroring the production on_standby_connected -> stale endpoint eviction -> later-batched io_handler() read chain.

Fix

FIXED (2026-07-18). src/lib/netio.c: extracted io_worker's per-batch dispatch into a new static io_worker_dispatch_batch(struct epoll_event*, int) (~line 639-654), called from io_worker's while(1) loop in place of the old inline for-loop. It now takes an endpoint_ref() on every endpoint in the batch BEFORE dispatching any of them, dispatches all events via io_handler() as before, then endpoint_unref()s every endpoint only AFTER the whole batch has been processed -- mirroring the SAME endpoint_ref/endpoint_unref bracket idiom this codebase already uses for the vtask dispatch path (bug-1298) and INVOKE_MSG_HANDLER (netio.h), applied here to io_worker's own epoll dispatch loop, the one dispatcher that didn't yet have it. With this bracket, on_standby_connected's endpoint_unref() on a stale duplicate can only drop that endpoint's refcount from 2 (owner + io_worker's batch ref) to 1 during the batch -- never to 0 -- so the free is deferred until io_worker's own trailing unref runs after the ENTIRE batch (including that endpoint's own later event) has been safely dispatched. ref/unref are the existing lock-free _Atomic-int operations (netio.c endpoint_ref/endpoint_unref) with no I/O and no lock acquisition, so this adds zero blocking risk to the io_worker hot path (satisfies the async-io steering rule against blocking in vtask/event-loop handlers). Also added a small always-compiled test seam, io_worker_dispatch_batch_for_test() (netio.c + netio.h) -- not XCE_TESTING-gated because netio.c compiles once into libxce.a with no per-target defines (unlike controller/ha.c, which the controller test binary compiles directly with -DXCE_TESTING), so it follows this file's existing convention of unconditionally exporting its internals (endpoint_ref, endpoint_unref, endpoint_mark_tearing_down, etc. are all plain public functions already).

Regression test: src/lib/testsuite/test_netio_batch_dispatch_uaf.c (test_batch_dispatch_survives_cross_endpoint_eviction), registered in src/lib/meson.build and src/lib/testsuite/test.c. Drives io_worker_dispatch_batch_for_test() with a synthetic 2-event batch: event 0 -> an endpoint whose conn_handler calls endpoint_unref() on event 1's endpoint (simulating on_standby_connected's stale-duplicate eviction); event 1 -> an endpoint with refcnt==1 (its only ref is the one event 0's handler drops) whose conn_handler just records that it ran. TDD verified: RED confirmed pre-fix (ASAN heap-use-after-free at netio.c:587 in io_handler, object freed at netio.c:843 endpoint_unref from the injected handler -- captured verbatim before the fix was restored); GREEN confirmed post-fix (both handlers run safely, exit 0, re-confirmed again standalone after both live soaks completed). Full regression: libxce lib suite 294/0/0 (xcelib_test, ASAN, ~/xce.build), controller_unit_test 972/0/0 with zero AddressSanitizer/heap-use-after-free/heap-buffer-overflow/double-free/SEGV hits in the full test log. Live verification beyond the unit test: two independent fresh sequential 10-round /xce-test-ha HA soaks on real ext4 (/home/claude/tmp/xce-bug1447-verify-1: 10/10 rounds, 33 jobs, zero job loss, all 3 nodes led; /home/claude/tmp/xce-bug1447-verify-2: 10/10 rounds, 36 jobs, zero job loss, all 3 nodes led including a forced-election leadership-coverage check on ctrl-3) -- both fully clean, zero crashes, and an explicit grep of every node's logs in both runs for the bug-1447 signature (AddressSanitizer/DEADLYSIGNAL/SEGV/ssl3_read_bytes/handle_ssl_handshake) found zero matches.

Verification

Not recorded (migrated from buglog.json).

References

• Related: bug-1282, bug-1286, bug-1298
• Gitea: #32