myxce/.wolf/bugs/bug-1916.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits
bug-1916	fixed	2026-10-03		lib	crash	2026-10-03T16:48:14Z	2026-10-03T16:48:14Z	1	src/lib/netio.c src/lib/xcectx.c src/lib/client_failover.c src/lib/request.c	asan uaf failover client worker	bug-352 bug-610	0fb2e14c

bug-1916: ASAN heap-use-after-free in free_packet via endpoint_fini <- xce_ctx_reconnect on client failover

Description

Observed:

==729084==ERROR: AddressSanitizer: heap-use-after-free ... READ of size 8 ... thread T10
    #0 free_packet ../lib/netio.c:463
    #1 endpoint_fini ../lib/netio.c:1210
    #2 xce_ctx_reconnect ../lib/xcectx.c:366
    #3 xce_failover_rotate ../lib/client_failover.c:160
    #4 xcecli_retry_next ../lib/client_failover.c:388
    #5 xcecli_join ../lib/client.c:307
    #6 perform_pending_rejoin ../worker/src/controller_connector.c:442
freed by: xce_decode ../lib/request.c:284 <- vtask_entry ../lib/xcectx.c:139 (vtask thread)
allocated by: handle_input ../lib/netio.c:479 <- io_handler (iomgr)

The worker (xcecd_virtual, a libxce client) was rejoining after the primary was killed. It failed over to ctrl-2, which was still a standby and redirected the role-1 join to the dead old primary (handle_join_ standby redirecting ... to localhost:9991) and closed the endpoint. The client's failover rotation then ran xce_ctx_reconnect → endpoint_fini → free_packet, reading a packet that a vtask had already freed in xce_decode while decoding a reply on the same endpoint.

Expected: client failover tears down the old endpoint only after (or without racing) in-flight decode vtasks, as the controller-side teardown does since bug-352/bug-1839 (deferred, per-endpoint FIFO teardown). Impact: crash of any libxce client (worker, scheduler, xce_test, apiserver backend) during controller failover when a reply decode races the failover rotation; the worker aborts, so its jobs' rejoin-resync never happens. Pre-existing client-library race, independent of bug-1913 (no controller code in the stack). Found by: soak (/xce-test-ha sequential, 8 rounds, under 40 busy loops on 32 cores, validating bug-1913)

UPDATE (2026-10-03): Second loaded soak (2026-10-03, round 2): the same heap-use-after-free (free_packet <- endpoint_fini <- xce_ctx_reconnect <- xce_failover_rotate <- xcecli_join; freed by xce_decode on a vtask) hit BOTH the worker and the scheduler (scheduler.stderr), so every libxce client is exposed. Now 2/2 loaded soak runs. ASAN reports kept in the session scratchpad as uaf-worker-2.stderr / uaf-scheduler-2.stderr.

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ 6d9ef3d0-dirty
• Deployment: local 3-node HA (/xce-test-ha), one xcecd_virtual worker + scheduler, 40 yes busy loops on 32 cores

Reproduction

1. Start 40 yes >/dev/null busy loops on a 32-core host.
2. XCE_HA_ROUNDS=8 nix develop ./src --command bash .claude/skills/xce-test-ha/run-ha-soak.sh 8 <scratch dir>.
3. Round 1 fails worker never logged a fresh rejoin after the failover; logs/soak-worker.stderr holds the ASAN report above. The slow election under load (new primary 7.4 s after the kill) leaves the worker failing over to a still-standby node that redirects and closes the endpoint.

• Frequency: 2/2 runs, 2026-10-03 (loaded 8-round soaks; worker both times, scheduler once); 0 in unloaded 20-round sequential and 6-round chaotic soaks the same day
• Evidence: ASAN report kept at the session scratchpad uaf-worker.stderr; deploy dir /home/claude/tmp/xce-ha-load-1913/logs (ctrl-*.log, xcecd-soak-worker.log)

Root cause

Packet handoff without detach. vtask_handle_msg (src/lib/vtaskmgr.c ~606) copied ep->fsm_in.pkt into the VTASK_IO task and published it with vtask_enq, but left ep->fsm_in.pkt set; INVOKE_MSG_HANDLER (src/include/netio.h:148-158) clears it only after the handler returns. In that window a vcpu ran xcectx vtask_entry → xce_decode, which frees the packet (request.c:283-284), then woke the client caller; the caller's xcecli_join → failover rotation → xce_ctx_reconnect → endpoint_fini freed the dangling slot again (netio.c:1210 → free_packet netio.c:463). The io_worker must be preempted between vtask_enq and its post-handler NULL store, which is why only heavily loaded soaks hit it. The controller's peer-control path already took the packet and cleared the slot first (bug-352 pattern); the shared vtask_handle_msg never did, and the vtaskmgr.h comment calling the client-path packet endpoint-owned was wrong.

Fix

vtask_handle_msg sets ep->fsm_in.pkt = NULL before vtask_enq publishes the task and sets task->owns_io_pkt = 1, so the task is the packet's sole owner: a never-run task's packet is freed by taskq_item_killer at vtaskmgr_fini, and an enqueue failure frees packet, data and task (previously leaked). xcectx vtask_entry frees the packet on its early -EINVAL exits and NULLs data->pkt after xce_decode. Comments in vtaskmgr.h / vtaskmgr.c corrected. Both entries that receive such packets (xcectx vtask_entry, controller cluster.c myreq_handler) already consume them unconditionally via xce_decode. Fix commit 0fb2e14c. Remaining, separate: bug-1917 (reconnect vs io_worker race), bug-1918 (early-exit packet leak).

Verification

• Test: test_endpoint_fini_after_vtask_freed_pkt in src/lib/testsuite/test_vtask_pkt_handoff.c — RED before the fix: ASAN heap-use-after-free in free_packet (netio.c:463) ← endpoint_fini (netio.c:1210), freed by xce_decode (request.c:284), the soak's exact stack (reviewer reproduced it again with only the NULL-before-enqueue line reverted); GREEN after. Companions test_handoff_detaches_before_task_runs, test_handoff_rejects_incomplete_fsm.
• Suites: libxce 733/0; controller 5/5 (1285/0), worker 15/15, scheduler 97/97, harness 7/7 (asan); xce-build.sh all --incremental green, 2026-10-03.
• Live: /xce-test-ha sequential 8 rounds under 40 busy loops on 32 cores PASS (30 jobs, zero loss, 8 worker-resync jobs reconciled) with no ASAN report in any process; the same soak aborted on this bug 2/2 before the fix.

References

• Related: none