43 lines
3.8 KiB
Markdown
43 lines
3.8 KiB
Markdown
---
|
|
id: "bug-1840"
|
|
status: "cannot-reproduce"
|
|
status_updated: "2026-09-25"
|
|
status_note: "no ASAN repro found. Strongest code-level candidate fixed 2026-09-25: bug-1769 (sched_make_alter_allocation wrote obj1->alloc 136 bytes into a 24-byte heap chunk for any ALTER '{..}||{..}' job -- exactly the kind of out-of-bounds write that corrupts glibc fastbin metadata and surfaces later in an unrelated malloc_consolidate). Reopen to investigating if it recurs, ideally captured on an ASAN build."
|
|
component: "scheduler"
|
|
severity: "crash"
|
|
timestamp: "2026-09-22T02:35:00Z"
|
|
last_seen: "2026-09-22T02:35:00Z"
|
|
occurrences: 1
|
|
files: ["unknown -- heap corruption detected far from its cause"]
|
|
tags: ["scheduler", "heap-corruption", "malloc-consolidate", "netio", "ssl-handshake", "live-crash", "user-reported", "needs-asan-repro", "needs-diagnosis"]
|
|
related: ["bug-1839"]
|
|
repro_attempts: "0/1 ASAN run, 2026-09-25: 3-node HA + worker + scheduler, 8 waves x 6 concurrent 'jobsub -a 50' (2400 elements) on the current ASAN build -- no AddressSanitizer report from the scheduler; plus ~10 ASAN HA soaks today (sequential/chaotic, all with scheduler replay-verify) with no scheduler heap error"
|
|
legacy: true
|
|
gitea_issue: 74
|
|
gitea_synced_status: "cannot-reproduce"
|
|
gitea_synced_hash: "0bf94f05143e"
|
|
---
|
|
|
|
# bug-1840: Live 3-node HA cluster crash (same soak run as bug-1839): xce_scheduler SIGABRT, glibc…
|
|
|
|
## Description
|
|
Live 3-node HA cluster crash (same soak run as bug-1839): xce_scheduler SIGABRT, glibc `malloc_consolidate(): unaligned fastbin chunk detected`, surfacing during an OpenSSL TLS handshake's MAC-buffer growth: abort -> malloc_printerr -> malloc_consolidate -> _int_malloc -> __libc_malloc2 -> CRYPTO_malloc -> CRYPTO_clear_realloc -> BUF_MEM_grow_clean -> mem_write -> bwrite_conv -> BIO_write -> ssl3_finish_mac -> tls_get_message_body -> state_machine -> handle_ssl_handshake (netio.c:548) -> io_handler (netio.c:588) -> io_worker_dispatch_batch (netio.c:648) -> io_worker (netio.c:689).
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
- Attempts: 0/1 ASAN run, 2026-09-25: 3-node HA + worker + scheduler, 8 waves x 6 concurrent 'jobsub -a 50' (2400 elements) on the current ASAN build -- no AddressSanitizer report from the scheduler; plus ~10 ASAN HA soaks today (sequential/chaotic, all with scheduler replay-verify) with no scheduler heap error
|
|
|
|
## Root cause
|
|
NOT YET DIAGNOSED -- heap corruption detected far from its cause; a corrupted fastbin chunk was only found during an unrelated later malloc_consolidate() call, inside an OpenSSL TLS handshake on the scheduler's io_worker thread. The actual out-of-bounds write that corrupted the allocator's freelist could have happened in any thread, any amount of time earlier, and is not reconstructable from a non-ASAN build.dbg core dump -- there is no shadow memory, no poisoning, nothing to localize it to a specific allocation or write site. Confirmed directly via gdb against the real core file (/tmp/core.xce_scheduler, build cd9ecb41): the crashing frames' locals (malloc_consolidate's av/fb/p) are all optimized out / no debug symbols available for glibc or libcrypto/libssl in this environment, so there is nothing further to extract from the core itself beyond the symbol-only backtrace already captured. Needs an ASAN reproduction of the same 3-node HA deploy soak scenario to catch the actual out-of-bounds write with a real stack trace.
|
|
|
|
## Fix
|
|
NOT FIXED -- awaiting an ASAN repro. Recommended next step: `/xce-build scheduler asan` (or `bash src/tools/xce-build.sh scheduler asan`), then rerun the same HA deploy scenario that produced /tmp/core.xce_scheduler until ASAN aborts at the real corruption site, which will pinpoint the actual overflow/write instead of just its downstream symptom.
|
|
|
|
## Verification
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## References
|
|
- Related: bug-1839
|
|
- Gitea: #74
|