myxce/.wolf/bugs/bug-1917.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1917	fixed	2026-10-04		lib	crash	2026-10-03T22:55:30Z	2026-10-03T22:55:30Z	1	src/lib/xcectx.c src/lib/netio.c	race failover client ssl	bug-1916	431a6d70 f26a2d45	342	fixed	8e6a90f1f914

bug-1917: xce_ctx_reconnect re-inits the endpoint on the caller thread while the io_worker may still use it

Description

Observed: Not yet observed as a crash -- found by code review while fixing bug-1916. xce_ctx_reconnect (src/lib/xcectx.c ~366) calls endpoint_fini / endpoint_init on the client's caller thread for an endpoint embedded in the context, while the io_worker may still be dispatching the same endpoint (still monitored). Exposed operations: the io_worker's post-handler store into ep->fsm_in.pkt and endpoint_unref (harmless after the bug-1916 fix), a concurrent SSL_read / handle_error on the old fd/SSL against SSL_free / close, and fsm_out.pkt freed while handle_output is writing it. vtask_entry also reads ep->owner without a lock while reconnect rewrites it.

Expected: reconnect first unmonitors the old endpoint and waits for the io_worker to finish any in-flight dispatch on it (quiesce), then tears it down, as the controller side does with deferred per-endpoint teardown (bug-1839). Impact: potential use-after-free / crash in OpenSSL or handle_output stacks under xce_ctx_reconnect during client failover (worker, scheduler, xce_test, apiserver backend), most likely under heavy CPU load, like bug-1916. Found by: review (bug-1916 fix, 2026-10-03)

UPDATE (2026-10-03): Review of the bug-1916 fix (2026-10-03) adds a second hazard: a reply task from the OLD connection still queued when xce_ctx_reconnect re-arms ep->owner can run against the NEW session (ep->owner read unsynchronised in xcectx.c vtask_entry ~134-135).

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ 6d9ef3d0-dirty
• Deployment: any libxce client failing over during a controller failover (seen context: /xce-test-ha under 40 busy loops on 32 cores)

Reproduction

1. Not reproduced yet. Candidate: loaded /xce-test-ha soaks (40 busy loops on 32 cores) with the bug-1916 fix in place; look for ASAN reports with xce_ctx_reconnect in the stack but not free_packet (e.g. SSL_read/SSL_free or handle_output frames).

• Frequency: 0/2 runs, 2026-10-03 (code-review finding; the 2 loaded soaks crashed on bug-1916 first)
• Evidence: code reading of src/lib/xcectx.c xce_ctx_reconnect and src/lib/netio.c endpoint_fini / handle_input / handle_output

Root cause

xce_ctx_reconnect (src/lib/xcectx.c) ran endpoint_fini/endpoint_init on ctx->ep straight away, while work dispatched for the old connection could still be running: the io_worker inside its epoll batch (it holds an endpoint reference for the whole batch: SSL_read/handle_output on the fd and SSL being freed), and VTASK_IO reply tasks still queued on the context's pool (each holds a reference and reads ep->owner without a lock). A queued reply task of the old connection that ran after the reconnect found ep->owner re-armed to the context and was delivered to the new session.

Fix

xce_ctx_reconnect first calls xce_ctx_quiesce_ep: unmonitor the old fd (no further io_worker events), clear ep->owner under ep->lock (stale reply tasks are dropped; vtask_entry frees their packet), then wait until the endpoint's reference count is back at its base value (0 for the embedded client endpoint, 1 for a heap one), polling every 1 ms. The wait is bounded at 2000 ms: if the caller is itself a task of this endpoint its own reference never drains, and the reconnect then proceeds as before with a WARN ("reconnecting anyway") rather than deadlocking. vtask_entry now reads ep->owner under ep->lock.

UPDATE (2026-10-08): Follow-up (2026-10-08, f26a2d45): a reconnect made from inside a task of the endpoint it re-establishes no longer waits out the 2 s bound (vtaskmgr tracks the endpoint whose VTASK_IO task the thread runs; xce_ctx_quiesce_ep returns at once in that case), and old-connection tasks queued behind it can no longer reach the new session: each VTASK_IO task records ep->reconnect_gen when queued and vtask_entry drops a task with a stale generation.

Verification

• Test: test_reconnect_drains_old_connection_tasks (new src/lib/testsuite/test_ctx_reconnect_quiesce.c): reconnect to a local listener, queue an old-connection packet via INVOKE_MSG_HANDLER -> handle_msg with the pool stopped, reconnect again while a helper starts the pool 100 ms in. RED before the fix (the stale task reached the context's request handler: expected 0 calls, was 1), GREEN and LSan-clean after.
• Suites: xce-build.sh all asan, 127/127 meson suites OK, Go tests ok (2026-10-04).
• Live: loaded /xce-test-ha sequential 30 rounds (40 busy loops on 32 cores), 2026-10-04: PASS, zero job loss, all 3 nodes led, no ASAN report in any controller/worker/scheduler log; client failover reconnects in worker/scheduler/xce_test throughout, and the quiesce never needed its timeout (no "reconnecting anyway" WARN).

UPDATE (2026-10-05): Live observation (2026-10-05, bug-1736 E2E): the xce_test fake worker hit the bounded-wait path -- a vtask handler of its own connection timed out waiting for a reply queued behind it (-110), failed over through xce_ctx_reconnect, and the quiesce logged 'old connection still has 2 in-flight reference(s) after 2000 ms; reconnecting anyway' and proceeded, as designed; no crash or ASAN report.

UPDATE (2026-10-08): Follow-up: test_reconnect_from_own_task_does_not_wait_or_leak (test_ctx_reconnect_quiesce.c) -- a reconnect from the endpoint's own task with a stale message queued behind it: RED before (took the full 2 s bound), GREEN after (under 500 ms, the queued stale message never reaches the handler), LSan clean; xce-build all asan 127/127 (2026-10-07).

References

• Related: none