fix(controller): standby-endpoint err handler freed ep->data under a live vtask (bug-1839)

Live 3-node HA soak crashed xce_controller: SIGSEGV in
peer_ctrl_msg_handler_inline (ha.c:7035) dereferencing a NULL
ep->data. gdb analysis of the actual core dump showed the endpoint
itself fully alive (refcnt=191964, tearing_down=0) while ep->data had
been synchronously freed and nulled by standby_ep_err_handler_primary,
the endpoint's err_handler, with zero coordination with the vtask pool
that was concurrently dispatching a peer-control message for the same
endpoint (g_drive_async_peerctrl_dispatch_enabled defaults ON).

The sibling handlers (vote_ep_err_handler, pre_vote_ep_err_handler)
already avoid this exact hazard, with an explicit comment: "Do NOT
free vctx here -- peer_ctrl_msg_handler may still reference ep->data."
standby_ep_err_handler_primary was never updated to the same
discipline when the vtask-pool dispatch machinery (bug-1197/bug-1298)
was introduced.

Fix: route standby_ep_err_handler_primary through the same
already-tested vote_ep_teardown_async() deferred, per-endpoint-FIFO-
serialized teardown that vote-endpoint demotion already uses, plus an
idempotency guard (endpoint_is_tearing_down) since the old "ep->data
== NULL" reentry check no longer applies once the free is deferred.
Also fixes a second-order double-unref this surfaced in
on_standby_connected's stale-duplicate eviction path, caught by ASAN.

TDD: two new tests prove the teardown is deferred until an in-flight
same-ep vtask completes, and that a repeated error callback on the
same endpoint is a safe no-op. Full controller suite: 1219/1219, zero
regressions.

Also logs bug-1840 (a separate scheduler heap-corruption crash from
the same soak run) as not-fixed pending an ASAN reproduction -- a
non-instrumented core dump cannot localize glibc's malloc_consolidate
abort back to the actual out-of-bounds write.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-22 03:21:14 +00:00
parent 4f94638a1d
commit e1da9f1e9b
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
7 changed files with 299 additions and 38 deletions

View file

@ -1,13 +1,13 @@
{
"version": 1,
"meta": {
"lastScanned": "2026-09-21T11:41:46.809Z",
"fileCount": 670,
"lastScanned": "2026-09-22T03:01:53.188Z",
"fileCount": 671,
"hits": 0,
"misses": 0,
"renderedHash": "9704b1ab4907098decf513d0818bd0fb1c4f301de23b9de49ce6882672c6da25",
"storeUpdatedAt": "2026-09-21T11:41:46.817Z",
"rootHash": "e3297d48be0b28ab"
"renderedHash": "02614f3b797e44059a670011f293e3047a15c20c2c31f6bcec3e18336f1f4f2d",
"storeUpdatedAt": "2026-09-22T03:01:53.196Z",
"rootHash": "e7f2dc076a00fe7b"
},
"files": {
".agent-shell/transcripts/2026-08-11-11-39-59.md": {
@ -10457,11 +10457,11 @@
},
"src/controller/ha.c": {
"description": "define _GNU_SOURCE",
"tokens": 127669,
"hash": "257355b271b99422",
"size": 483416,
"mtimeMs": 1789537740512.282,
"updatedAt": "2026-09-16T05:49:00.563Z",
"tokens": 127991,
"hash": "9278ac07e8071f45",
"size": 484628,
"mtimeMs": 1790046113136.1223,
"updatedAt": "2026-09-22T03:01:53.188Z",
"source": "hook"
},
"src/include/request.h": {
@ -10502,11 +10502,11 @@
},
"src/controller/testsuite/test_ha_liveness.c": {
"description": "Declares directly",
"tokens": 25885,
"hash": "183986d90eef9326",
"size": 99484,
"mtimeMs": 1789534901696.1792,
"updatedAt": "2026-09-16T05:01:41.736Z",
"tokens": 26012,
"hash": "c020980138115d54",
"size": 99964,
"mtimeMs": 1790045648548.2148,
"updatedAt": "2026-09-22T02:54:08.589Z",
"source": "hook"
},
"src/controller/testsuite/test_ha_heartbeat_committed.c": {
@ -12252,6 +12252,15 @@
"mtimeMs": 1789989591378.6973,
"updatedAt": "2026-09-21T11:19:51.419Z",
"source": "hook"
},
"src/controller/testsuite/test_ha_vep_teardown.c": {
"description": "",
"tokens": 8206,
"hash": "371d908c0b62aafa",
"size": 31661,
"mtimeMs": 1790044881003.8472,
"updatedAt": "2026-09-22T02:41:21.052Z",
"source": "hook"
}
}
}