fix(controller): standby-endpoint err handler freed ep->data under a live vtask (bug-1839)
Live 3-node HA soak crashed xce_controller: SIGSEGV in peer_ctrl_msg_handler_inline (ha.c:7035) dereferencing a NULL ep->data. gdb analysis of the actual core dump showed the endpoint itself fully alive (refcnt=191964, tearing_down=0) while ep->data had been synchronously freed and nulled by standby_ep_err_handler_primary, the endpoint's err_handler, with zero coordination with the vtask pool that was concurrently dispatching a peer-control message for the same endpoint (g_drive_async_peerctrl_dispatch_enabled defaults ON). The sibling handlers (vote_ep_err_handler, pre_vote_ep_err_handler) already avoid this exact hazard, with an explicit comment: "Do NOT free vctx here -- peer_ctrl_msg_handler may still reference ep->data." standby_ep_err_handler_primary was never updated to the same discipline when the vtask-pool dispatch machinery (bug-1197/bug-1298) was introduced. Fix: route standby_ep_err_handler_primary through the same already-tested vote_ep_teardown_async() deferred, per-endpoint-FIFO- serialized teardown that vote-endpoint demotion already uses, plus an idempotency guard (endpoint_is_tearing_down) since the old "ep->data == NULL" reentry check no longer applies once the free is deferred. Also fixes a second-order double-unref this surfaced in on_standby_connected's stale-duplicate eviction path, caught by ASAN. TDD: two new tests prove the teardown is deferred until an in-flight same-ep vtask completes, and that a repeated error callback on the same endpoint is a safe no-op. Full controller suite: 1219/1219, zero regressions. Also logs bug-1840 (a separate scheduler heap-corruption crash from the same soak run) as not-fixed pending an ASAN reproduction -- a non-instrumented core dump cannot localize glibc's malloc_consolidate abort back to the actual out-of-bounds write. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
4f94638a1d
commit
e1da9f1e9b
7 changed files with 299 additions and 38 deletions
|
|
@ -1,13 +1,13 @@
|
|||
{
|
||||
"version": 1,
|
||||
"meta": {
|
||||
"lastScanned": "2026-09-21T11:41:46.809Z",
|
||||
"fileCount": 670,
|
||||
"lastScanned": "2026-09-22T03:01:53.188Z",
|
||||
"fileCount": 671,
|
||||
"hits": 0,
|
||||
"misses": 0,
|
||||
"renderedHash": "9704b1ab4907098decf513d0818bd0fb1c4f301de23b9de49ce6882672c6da25",
|
||||
"storeUpdatedAt": "2026-09-21T11:41:46.817Z",
|
||||
"rootHash": "e3297d48be0b28ab"
|
||||
"renderedHash": "02614f3b797e44059a670011f293e3047a15c20c2c31f6bcec3e18336f1f4f2d",
|
||||
"storeUpdatedAt": "2026-09-22T03:01:53.196Z",
|
||||
"rootHash": "e7f2dc076a00fe7b"
|
||||
},
|
||||
"files": {
|
||||
".agent-shell/transcripts/2026-08-11-11-39-59.md": {
|
||||
|
|
@ -10457,11 +10457,11 @@
|
|||
},
|
||||
"src/controller/ha.c": {
|
||||
"description": "define _GNU_SOURCE",
|
||||
"tokens": 127669,
|
||||
"hash": "257355b271b99422",
|
||||
"size": 483416,
|
||||
"mtimeMs": 1789537740512.282,
|
||||
"updatedAt": "2026-09-16T05:49:00.563Z",
|
||||
"tokens": 127991,
|
||||
"hash": "9278ac07e8071f45",
|
||||
"size": 484628,
|
||||
"mtimeMs": 1790046113136.1223,
|
||||
"updatedAt": "2026-09-22T03:01:53.188Z",
|
||||
"source": "hook"
|
||||
},
|
||||
"src/include/request.h": {
|
||||
|
|
@ -10502,11 +10502,11 @@
|
|||
},
|
||||
"src/controller/testsuite/test_ha_liveness.c": {
|
||||
"description": "Declares directly",
|
||||
"tokens": 25885,
|
||||
"hash": "183986d90eef9326",
|
||||
"size": 99484,
|
||||
"mtimeMs": 1789534901696.1792,
|
||||
"updatedAt": "2026-09-16T05:01:41.736Z",
|
||||
"tokens": 26012,
|
||||
"hash": "c020980138115d54",
|
||||
"size": 99964,
|
||||
"mtimeMs": 1790045648548.2148,
|
||||
"updatedAt": "2026-09-22T02:54:08.589Z",
|
||||
"source": "hook"
|
||||
},
|
||||
"src/controller/testsuite/test_ha_heartbeat_committed.c": {
|
||||
|
|
@ -12252,6 +12252,15 @@
|
|||
"mtimeMs": 1789989591378.6973,
|
||||
"updatedAt": "2026-09-21T11:19:51.419Z",
|
||||
"source": "hook"
|
||||
},
|
||||
"src/controller/testsuite/test_ha_vep_teardown.c": {
|
||||
"description": "",
|
||||
"tokens": 8206,
|
||||
"hash": "371d908c0b62aafa",
|
||||
"size": 31661,
|
||||
"mtimeMs": 1790044881003.8472,
|
||||
"updatedAt": "2026-09-22T02:41:21.052Z",
|
||||
"source": "hook"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
# anatomy.md
|
||||
|
||||
> Auto-maintained by OpenWolf. Last scanned: 2026-09-21T11:41:46.809Z
|
||||
> Files: 670 tracked | Anatomy hits: 0 | Misses: 0
|
||||
> Auto-maintained by OpenWolf. Last scanned: 2026-09-22T03:01:53.188Z
|
||||
> Files: 671 tracked | Anatomy hits: 0 | Misses: 0
|
||||
|
||||
## ./
|
||||
|
||||
|
|
@ -884,7 +884,7 @@
|
|||
## src/controller/
|
||||
|
||||
- `ctrlstatus.c` — include "config.h" (~4588 tok)
|
||||
- `ha.c` — define _GNU_SOURCE (~127669 tok)
|
||||
- `ha.c` — define _GNU_SOURCE (~127991 tok)
|
||||
- `ha.h` — ifndef HA_H (~34444 tok)
|
||||
- `meson.build` (~3405 tok)
|
||||
|
||||
|
|
@ -895,9 +895,10 @@
|
|||
- `test_ctrlstatus.c` — include <stdatomic.h> (~12594 tok)
|
||||
- `test_e2e_capnp.c` — include <stdint.h> (~20187 tok)
|
||||
- `test_ha_heartbeat_committed.c` — include <stdatomic.h> (~5775 tok)
|
||||
- `test_ha_liveness.c` — Declares directly (~25885 tok)
|
||||
- `test_ha_liveness.c` — Declares directly (~26012 tok)
|
||||
- `test_ha_partition.c` (~5500 tok)
|
||||
- `test_ha_role.c` — include <signal.h> (~13998 tok)
|
||||
- `test_ha_vep_teardown.c` (~8206 tok)
|
||||
- `test_rcfile_integration.sh` — test_rcfile_integration.sh — xce-test-rc-file integration test, shell-driven (~4117 tok)
|
||||
|
||||
## src/harness/
|
||||
|
|
|
|||
|
|
@ -24872,6 +24872,62 @@
|
|||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-21T02:40:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1839",
|
||||
"timestamp": "2026-09-22T02:30:00Z",
|
||||
"error_message": "Live 3-node HA cluster crash: xce_controller SIGSEGV in peer_ctrl_msg_handler_inline (ha.c:7035, `ha_state_t *ha = vctx->ha;`). gdb analysis of the actual core dump (build cd9ecb41) confirmed vctx == NULL (i.e. ep->data == NULL) while the endpoint itself was fully alive and in active use: ep->refcnt=191964, ep->tearing_down=0, ep->heap_alloc=1, ep->msg_handler=peer_ctrl_msg_handler, ep->err_handler=standby_ep_err_handler_primary, ep->fsm_out.state=FSM_STATE_ERROR. The crash fired on the vtask-pool path (vcpu_entry -> run_task -> peer_ctrl_vtask_entry -> peer_ctrl_msg_handler_inline); g_drive_async_peerctrl_dispatch_enabled defaults to 1 (ON, the production path).",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "standby_ep_err_handler_primary (the err_handler for an outbound standby/heartbeat replication channel, registered in ha_connect_prep_peer) synchronously did `free(sctx); ep->data = NULL;` on whichever thread handle_error() called it from (the io_worker thread), with zero coordination with the vtask pool. If a peer-control vtask for the SAME ep was already queued or in flight (entirely plausible -- the endpoint is actively receiving replication ACKs/heartbeats via peer_ctrl_msg_handler -> peer_ctrl_dispatch_to_pool -> the vtask pool, and can independently hit a write/SSL error on that same connection), that vtask later ran peer_ctrl_msg_handler_inline, read ep->data as NULL, and crashed with a NULL dereference (not a use-after-free -- ep->data was explicitly nulled, not freed-and-reused memory). This is exactly the hazard this codebase already identified and fixed for the sibling vote_ep_ctx_t case: vote_ep_err_handler (ha.c:7787) and pre_vote_ep_err_handler (ha.c:7825) both explicitly do NOT free their context, with an inline comment ('Do NOT free vctx here -- peer_ctrl_msg_handler may still reference ep->data.'). standby_ep_err_handler_primary was never updated to the same discipline when the vtask-pool dispatch machinery (raft-drive-async, bug-1197/bug-1298) was introduced and retrofitted onto the other err handlers -- it slipped through because it's a distinct, older handler for the ongoing standby-heartbeat connection rather than the vote-time connection. Audited the other ~6 `ep->data = NULL` sites in ha.c (ha_connect_prep_peer/ha_connect_finish_peer/ha_connect_to_peer) and confirmed none of them share this bug class -- each frees an endpoint that either was never registered with iomgr at all, or whose iomgr_monitor() call itself just failed, so no vtask could ever have been dispatched against it.",
|
||||
"fix": "FIXED (2026-09-22, TDD, user-reported live crash). Reused the established, already-tested vote_ep_teardown_async(ha, ep) mechanism (ha.c:7232) instead of inventing a new one: it calls endpoint_mark_tearing_down(ep) (sets the flag + drains dispatch_inflight, closing the TOCTOU window against peer_ctrl_msg_handler's late-event gate), iomgr_unmonitor() if monitored, then enqueues a VTASK_IO teardown task bound to this ep via vtask_enq_classified -- the per-endpoint FIFO guarantees this teardown vtask runs strictly after any earlier-queued or in-flight same-ep peer-control vtask. Its body (vote_ep_teardown_vtask_entry) is fully generic over what ep->data holds -- it just endpoint_unref()s the endpoint, and endpoint_fini()'s own FREEUP(ep->data) frees whatever ep->data currently points to once the last reference truly releases. standby_ep_err_handler_primary's `free(sctx); ep->data = NULL;` tail was replaced with a single `vote_ep_teardown_async(ha, ep);` call (its ha->standby_endpoints removal and repl_remove_standby bookkeeping stayed unchanged, since those only read sctx's still-valid fields before the deferred free). Added an `if (endpoint_is_tearing_down(ep)) return 0;` idempotency guard at the top of the handler: the old `if (!sctx) return 0` accidentally doubled as a reentry guard (a second call after the first had already nulled ep->data would bail), but that protection disappears once ep->data is no longer nulled synchronously, and handle_error() firing this handler more than once for the same ep (e.g. a read error and a write error both observed before iomgr_unmonitor takes effect) would otherwise double-drop the endpoint's owner reference. Also found and fixed a second-order regression this exposed: on_standby_connected's stale-duplicate-channel eviction path (ha.c ~9461-9476) called standby_ep_err_handler_primary(stale_same_peer) and THEN did its own iomgr_unmonitor + endpoint_unref(stale_same_peer) afterward -- correct under the OLD contract (the handler used to only free ep->data, leaving ref-dropping to the caller) but now a double-unref/use-after-free once the handler owns the full teardown itself. Removed the caller's now-redundant iomgr_unmonitor/endpoint_unref calls; the err handler's own vote_ep_teardown_async call handles everything. Caught by ASAN (heap-use-after-free in endpoint_unref via on_standby_connected, immediately on the first full-suite rebuild after the primary fix) rather than by design review -- confirms the value of running the full suite, not just the new tests, after this kind of shared-contract change. Also updated 3 pre-existing tests in test_ha_liveness.c (test_reconnect_resolves_dual_channel_same_peer, test_reconnect_resolves_dual_channel_promoted_vote_ep, test_reconnect_preserves_other_peer_channel) that called standby_ep_err_handler_primary_for_test(ep) followed by a manual endpoint_fini(ep)/free(ep) -- with ha.pool == NULL in these unit fixtures, vote_ep_teardown_async now takes its documented no-pool inline fallback (WARN + endpoint_unref, refcnt 1->0, fini+free) inside the handler itself, so the tests' own manual cleanup became a double-free; removed it. TDD: two new tests in test_ha_vep_teardown.c using a real standby_ep_ctx_t (standby_ep_ctx_build_for_test) and the real standby_ep_err_handler_primary_for_test entry point (not the generic vote_ep_ctx_t stand-in test (a) in that file uses): test_standby_err_handler_defers_teardown_until_inflight_completes proves the deferred-teardown ordering (mirrors test_vep_teardown_runs_after_inflight's gate/sequence-ticket technique -- parks an in-flight same-ep vtask, calls the err handler, asserts teardown does NOT run while the vtask is parked, then releases it and asserts teardown runs strictly after); test_standby_err_handler_idempotent_against_double_invocation proves calling the handler twice in a row on the same (still-alive, tearing-down) ep is a safe no-op with exactly one teardown vtask ever running. RED confirmed against the pre-fix handler: both new tests failed with 'ep must be marked tearing_down...' (endpoint_is_tearing_down never became true, since the old code never called vote_ep_teardown_async at all) -- 1219 tests, 2 failures, zero other regressions. GREEN after the fix plus the on_standby_connected correction: full controller suite 1219/1219 Ok, zero failures.",
|
||||
"tags": [
|
||||
"controller",
|
||||
"ha",
|
||||
"standby-endpoint",
|
||||
"err-handler",
|
||||
"null-deref",
|
||||
"vtask-pool",
|
||||
"peer-ctrl-dispatch",
|
||||
"vote-ep-teardown",
|
||||
"related-to-bug-1197",
|
||||
"related-to-bug-1298",
|
||||
"live-crash",
|
||||
"user-reported",
|
||||
"fixed",
|
||||
"tdd"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1197",
|
||||
"bug-1298"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-22T02:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1840",
|
||||
"timestamp": "2026-09-22T02:35:00Z",
|
||||
"error_message": "Live 3-node HA cluster crash (same soak run as bug-1839): xce_scheduler SIGABRT, glibc `malloc_consolidate(): unaligned fastbin chunk detected`, surfacing during an OpenSSL TLS handshake's MAC-buffer growth: abort -> malloc_printerr -> malloc_consolidate -> _int_malloc -> __libc_malloc2 -> CRYPTO_malloc -> CRYPTO_clear_realloc -> BUF_MEM_grow_clean -> mem_write -> bwrite_conv -> BIO_write -> ssl3_finish_mac -> tls_get_message_body -> state_machine -> handle_ssl_handshake (netio.c:548) -> io_handler (netio.c:588) -> io_worker_dispatch_batch (netio.c:648) -> io_worker (netio.c:689).",
|
||||
"file": "unknown -- heap corruption detected far from its cause",
|
||||
"root_cause": "NOT YET DIAGNOSED -- heap corruption detected far from its cause; a corrupted fastbin chunk was only found during an unrelated later malloc_consolidate() call, inside an OpenSSL TLS handshake on the scheduler's io_worker thread. The actual out-of-bounds write that corrupted the allocator's freelist could have happened in any thread, any amount of time earlier, and is not reconstructable from a non-ASAN build.dbg core dump -- there is no shadow memory, no poisoning, nothing to localize it to a specific allocation or write site. Confirmed directly via gdb against the real core file (/tmp/core.xce_scheduler, build cd9ecb41): the crashing frames' locals (malloc_consolidate's av/fb/p) are all optimized out / no debug symbols available for glibc or libcrypto/libssl in this environment, so there is nothing further to extract from the core itself beyond the symbol-only backtrace already captured. Needs an ASAN reproduction of the same 3-node HA deploy soak scenario to catch the actual out-of-bounds write with a real stack trace.",
|
||||
"fix": "NOT FIXED -- awaiting an ASAN repro. Recommended next step: `/xce-build scheduler asan` (or `bash src/tools/xce-build.sh scheduler asan`), then rerun the same HA deploy scenario that produced /tmp/core.xce_scheduler until ASAN aborts at the real corruption site, which will pinpoint the actual overflow/write instead of just its downstream symptom.",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"heap-corruption",
|
||||
"malloc-consolidate",
|
||||
"netio",
|
||||
"ssl-handshake",
|
||||
"live-crash",
|
||||
"user-reported",
|
||||
"needs-asan-repro",
|
||||
"not-fixed",
|
||||
"unfixed",
|
||||
"needs-diagnosis"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1839"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-22T02:35:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5626,3 +5626,11 @@
|
|||
| 04:26 | Edited src/scheduler/src/sim_loop.c | added 1 condition(s) | ~128 |
|
||||
| 04:33 | Edited src/scheduler/src/sim_loop.c | modified sim_reconcile_invoke_on_dispatch() | ~90 |
|
||||
| 04:41 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/design.md | modified Correction() | ~557 |
|
||||
| 19:40 | Edited src/controller/testsuite/test_ha_vep_teardown.c | 4→9 lines | ~159 |
|
||||
| 19:41 | Edited src/controller/testsuite/test_ha_vep_teardown.c | added 1 condition(s) | ~2024 |
|
||||
| 19:41 | Edited src/controller/testsuite/test_ha_vep_teardown.c | poll_until_ge() → poll_until() | ~41 |
|
||||
| 19:53 | Edited src/controller/ha.c | added 1 condition(s) | ~558 |
|
||||
| 19:53 | Edited src/controller/testsuite/test_ha_liveness.c | 8→11 lines | ~178 |
|
||||
| 19:53 | Edited src/controller/testsuite/test_ha_liveness.c | 4→4 lines | ~60 |
|
||||
| 19:54 | Edited src/controller/testsuite/test_ha_liveness.c | 6→4 lines | ~69 |
|
||||
| 20:01 | Edited src/controller/ha.c | modified XCE_TESTING() | ~235 |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue