3.8 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | repro_attempts | legacy | gitea_issue | gitea_synced_status | gitea_synced_hash | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1840 | cannot-reproduce | 2026-09-25 | no ASAN repro found. Strongest code-level candidate fixed 2026-09-25: bug-1769 (sched_make_alter_allocation wrote obj1->alloc 136 bytes into a 24-byte heap chunk for any ALTER '{..}||{..}' job -- exactly the kind of out-of-bounds write that corrupts glibc fastbin metadata and surfaces later in an unrelated malloc_consolidate). Reopen to investigating if it recurs, ideally captured on an ASAN build. | scheduler | crash | 2026-09-22T02:35:00Z | 2026-09-22T02:35:00Z | 1 |
|
|
|
0/1 ASAN run, 2026-09-25: 3-node HA + worker + scheduler, 8 waves x 6 concurrent 'jobsub -a 50' (2400 elements) on the current ASAN build -- no AddressSanitizer report from the scheduler; plus ~10 ASAN HA soaks today (sequential/chaotic, all with scheduler replay-verify) with no scheduler heap error | true | 74 | cannot-reproduce | 0bf94f05143e |
bug-1840: Live 3-node HA cluster crash (same soak run as bug-1839): xce_scheduler SIGABRT, glibc…
Description
Live 3-node HA cluster crash (same soak run as bug-1839): xce_scheduler SIGABRT, glibc malloc_consolidate(): unaligned fastbin chunk detected, surfacing during an OpenSSL TLS handshake's MAC-buffer growth: abort -> malloc_printerr -> malloc_consolidate -> _int_malloc -> __libc_malloc2 -> CRYPTO_malloc -> CRYPTO_clear_realloc -> BUF_MEM_grow_clean -> mem_write -> bwrite_conv -> BIO_write -> ssl3_finish_mac -> tls_get_message_body -> state_machine -> handle_ssl_handshake (netio.c:548) -> io_handler (netio.c:588) -> io_worker_dispatch_batch (netio.c:648) -> io_worker (netio.c:689).
Environment
Not recorded (migrated from buglog.json).
Reproduction
- Attempts: 0/1 ASAN run, 2026-09-25: 3-node HA + worker + scheduler, 8 waves x 6 concurrent 'jobsub -a 50' (2400 elements) on the current ASAN build -- no AddressSanitizer report from the scheduler; plus ~10 ASAN HA soaks today (sequential/chaotic, all with scheduler replay-verify) with no scheduler heap error
Root cause
NOT YET DIAGNOSED -- heap corruption detected far from its cause; a corrupted fastbin chunk was only found during an unrelated later malloc_consolidate() call, inside an OpenSSL TLS handshake on the scheduler's io_worker thread. The actual out-of-bounds write that corrupted the allocator's freelist could have happened in any thread, any amount of time earlier, and is not reconstructable from a non-ASAN build.dbg core dump -- there is no shadow memory, no poisoning, nothing to localize it to a specific allocation or write site. Confirmed directly via gdb against the real core file (/tmp/core.xce_scheduler, build cd9ecb41): the crashing frames' locals (malloc_consolidate's av/fb/p) are all optimized out / no debug symbols available for glibc or libcrypto/libssl in this environment, so there is nothing further to extract from the core itself beyond the symbol-only backtrace already captured. Needs an ASAN reproduction of the same 3-node HA deploy soak scenario to catch the actual out-of-bounds write with a real stack trace.
Fix
NOT FIXED -- awaiting an ASAN repro. Recommended next step: /xce-build scheduler asan (or bash src/tools/xce-build.sh scheduler asan), then rerun the same HA deploy scenario that produced /tmp/core.xce_scheduler until ASAN aborts at the real corruption site, which will pinpoint the actual overflow/write instead of just its downstream symptom.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1839
- Gitea: #74