68 lines
6.4 KiB
Markdown
68 lines
6.4 KiB
Markdown
---
|
|
id: "bug-1916"
|
|
status: "fixed"
|
|
status_updated: "2026-10-03"
|
|
status_note: ""
|
|
component: "lib"
|
|
severity: "crash"
|
|
timestamp: "2026-10-03T16:48:14Z"
|
|
last_seen: "2026-10-03T16:48:14Z"
|
|
occurrences: 1
|
|
files: ["src/lib/netio.c", "src/lib/xcectx.c", "src/lib/client_failover.c", "src/lib/request.c"]
|
|
tags: ["asan", "uaf", "failover", "client", "worker"]
|
|
related: ["bug-352", "bug-610"]
|
|
fix_commits: ["0fb2e14c"]
|
|
---
|
|
|
|
# bug-1916: ASAN heap-use-after-free in free_packet via endpoint_fini <- xce_ctx_reconnect on client failover
|
|
|
|
## Description
|
|
**Observed:**
|
|
```
|
|
==729084==ERROR: AddressSanitizer: heap-use-after-free ... READ of size 8 ... thread T10
|
|
#0 free_packet ../lib/netio.c:463
|
|
#1 endpoint_fini ../lib/netio.c:1210
|
|
#2 xce_ctx_reconnect ../lib/xcectx.c:366
|
|
#3 xce_failover_rotate ../lib/client_failover.c:160
|
|
#4 xcecli_retry_next ../lib/client_failover.c:388
|
|
#5 xcecli_join ../lib/client.c:307
|
|
#6 perform_pending_rejoin ../worker/src/controller_connector.c:442
|
|
freed by: xce_decode ../lib/request.c:284 <- vtask_entry ../lib/xcectx.c:139 (vtask thread)
|
|
allocated by: handle_input ../lib/netio.c:479 <- io_handler (iomgr)
|
|
```
|
|
The worker (`xcecd_virtual`, a libxce client) was rejoining after the primary was killed. It failed over to ctrl-2, which was still a standby and redirected the role-1 join to the dead old primary (`handle_join_ standby redirecting ... to localhost:9991`) and closed the endpoint. The client's failover rotation then ran `xce_ctx_reconnect` → `endpoint_fini` → `free_packet`, reading a packet that a vtask had already freed in `xce_decode` while decoding a reply on the same endpoint.
|
|
|
|
**Expected:** client failover tears down the old endpoint only after (or without racing) in-flight decode vtasks, as the controller-side teardown does since bug-352/bug-1839 (deferred, per-endpoint FIFO teardown).
|
|
**Impact:** crash of any libxce client (worker, scheduler, xce_test, apiserver backend) during controller failover when a reply decode races the failover rotation; the worker aborts, so its jobs' rejoin-resync never happens. Pre-existing client-library race, independent of bug-1913 (no controller code in the stack).
|
|
**Found by:** soak (`/xce-test-ha` sequential, 8 rounds, under 40 busy loops on 32 cores, validating bug-1913)
|
|
|
|
UPDATE (2026-10-03): Second loaded soak (2026-10-03, round 2): the same heap-use-after-free (free_packet <- endpoint_fini <- xce_ctx_reconnect <- xce_failover_rotate <- xcecli_join; freed by xce_decode on a vtask) hit BOTH the worker and the scheduler (scheduler.stderr), so every libxce client is exposed. Now 2/2 loaded soak runs. ASAN reports kept in the session scratchpad as uaf-worker-2.stderr / uaf-scheduler-2.stderr.
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (xce.build)
|
|
- Code: scheduler-framework @ 6d9ef3d0-dirty
|
|
- Deployment: local 3-node HA (`/xce-test-ha`), one `xcecd_virtual` worker + scheduler, 40 `yes` busy loops on 32 cores
|
|
|
|
## Reproduction
|
|
1. Start 40 `yes >/dev/null` busy loops on a 32-core host.
|
|
2. `XCE_HA_ROUNDS=8 nix develop ./src --command bash .claude/skills/xce-test-ha/run-ha-soak.sh 8 <scratch dir>`.
|
|
3. Round 1 fails `worker never logged a fresh rejoin after the failover`; `logs/soak-worker.stderr` holds the ASAN report above. The slow election under load (new primary 7.4 s after the kill) leaves the worker failing over to a still-standby node that redirects and closes the endpoint.
|
|
- Frequency: 2/2 runs, 2026-10-03 (loaded 8-round soaks; worker both times, scheduler once); 0 in unloaded 20-round sequential and 6-round chaotic soaks the same day
|
|
- Evidence: ASAN report kept at the session scratchpad `uaf-worker.stderr`; deploy dir /home/claude/tmp/xce-ha-load-1913/logs (ctrl-*.log, xcecd-soak-worker.log)
|
|
|
|
## Root cause
|
|
Packet handoff without detach. `vtask_handle_msg` (src/lib/vtaskmgr.c ~606) copied `ep->fsm_in.pkt` into the VTASK_IO task and published it with `vtask_enq`, but left `ep->fsm_in.pkt` set; `INVOKE_MSG_HANDLER` (src/include/netio.h:148-158) clears it only after the handler returns. In that window a vcpu ran xcectx `vtask_entry` → `xce_decode`, which frees the packet (request.c:283-284), then woke the client caller; the caller's `xcecli_join` → failover rotation → `xce_ctx_reconnect` → `endpoint_fini` freed the dangling slot again (netio.c:1210 → `free_packet` netio.c:463). The io_worker must be preempted between `vtask_enq` and its post-handler NULL store, which is why only heavily loaded soaks hit it. The controller's peer-control path already took the packet and cleared the slot first (bug-352 pattern); the shared `vtask_handle_msg` never did, and the vtaskmgr.h comment calling the client-path packet endpoint-owned was wrong.
|
|
|
|
## Fix
|
|
`vtask_handle_msg` sets `ep->fsm_in.pkt = NULL` before `vtask_enq` publishes the task and sets `task->owns_io_pkt = 1`, so the task is the packet's sole owner: a never-run task's packet is freed by `taskq_item_killer` at `vtaskmgr_fini`, and an enqueue failure frees packet, data and task (previously leaked). xcectx `vtask_entry` frees the packet on its early -EINVAL exits and NULLs `data->pkt` after `xce_decode`. Comments in vtaskmgr.h / vtaskmgr.c corrected. Both entries that receive such packets (xcectx `vtask_entry`, controller cluster.c `myreq_handler`) already consume them unconditionally via `xce_decode`. Fix commit 0fb2e14c. Remaining, separate: bug-1917 (reconnect vs io_worker race), bug-1918 (early-exit packet leak).
|
|
|
|
## Verification
|
|
- Test: `test_endpoint_fini_after_vtask_freed_pkt` in `src/lib/testsuite/test_vtask_pkt_handoff.c` — RED before the fix: ASAN heap-use-after-free in `free_packet` (netio.c:463) ← `endpoint_fini` (netio.c:1210), freed by `xce_decode` (request.c:284), the soak's exact stack (reviewer reproduced it again with only the NULL-before-enqueue line reverted); GREEN after. Companions `test_handoff_detaches_before_task_runs`, `test_handoff_rejects_incomplete_fsm`.
|
|
- Suites: libxce 733/0; controller 5/5 (1285/0), worker 15/15, scheduler 97/97, harness 7/7 (asan); `xce-build.sh all --incremental` green, 2026-10-03.
|
|
- Live: `/xce-test-ha` sequential 8 rounds under 40 busy loops on 32 cores PASS (30 jobs, zero loss, 8 worker-resync jobs reconciled) with no ASAN report in any process; the same soak aborted on this bug 2/2 before the fix.
|
|
|
|
## References
|
|
- Related: none
|