docs(buglog): bug-1913 and bug-1916 fixed; log bug-1915, bug-1917, bug-1918

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-03 23:18:53 +00:00

3.2 KiB

Raw Blame History

status

status_updated

status_note

component

severity

timestamp

last_seen

occurrences

files

bug-1917: xce_ctx_reconnect re-inits the endpoint on the caller thread while the io_worker may still use it

Description

Observed: Not yet observed as a crash -- found by code review while fixing bug-1916. xce_ctx_reconnect (src/lib/xcectx.c ~366) calls endpoint_fini / endpoint_init on the client's caller thread for an endpoint embedded in the context, while the io_worker may still be dispatching the same endpoint (still monitored). Exposed operations: the io_worker's post-handler store into ep->fsm_in.pkt and endpoint_unref (harmless after the bug-1916 fix), a concurrent SSL_read / handle_error on the old fd/SSL against SSL_free / close, and fsm_out.pkt freed while handle_output is writing it. vtask_entry also reads ep->owner without a lock while reconnect rewrites it.

Expected: reconnect first unmonitors the old endpoint and waits for the io_worker to finish any in-flight dispatch on it (quiesce), then tears it down, as the controller side does with deferred per-endpoint teardown (bug-1839). Impact: potential use-after-free / crash in OpenSSL or handle_output stacks under xce_ctx_reconnect during client failover (worker, scheduler, xce_test, apiserver backend), most likely under heavy CPU load, like bug-1916. Found by: review (bug-1916 fix, 2026-10-03)

UPDATE (2026-10-03): Review of the bug-1916 fix (2026-10-03) adds a second hazard: a reply task from the OLD connection still queued when xce_ctx_reconnect re-arms ep->owner can run against the NEW session (ep->owner read unsynchronised in xcectx.c vtask_entry ~134-135).

Environment

OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
Build: asan (xce.build)
Code: scheduler-framework @ 6d9ef3d0-dirty
Deployment: any libxce client failing over during a controller failover (seen context: /xce-test-ha under 40 busy loops on 32 cores)

Reproduction

Not reproduced yet. Candidate: loaded /xce-test-ha soaks (40 busy loops on 32 cores) with the bug-1916 fix in place; look for ASAN reports with xce_ctx_reconnect in the stack but not free_packet (e.g. SSL_read/SSL_free or handle_output frames).

Frequency: 0/2 runs, 2026-10-03 (code-review finding; the 2 loaded soaks crashed on bug-1916 first)
Evidence: code reading of src/lib/xcectx.c xce_ctx_reconnect and src/lib/netio.c endpoint_fini / handle_input / handle_output

Root cause

TODO: required for fixed/partial

Fix