---
id: "bug-1699"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "crash"
timestamp: "2026-08-26T07:35:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c or src/controller/cluster.c (exact function not identified -- crash occurred somewhere in the ctrlstatus request-handling path shortly after raft_cb_become_follower/first_boot=1)"]
tags: ["controller", "ha", "segfault", "sigsegv", "unreproduced", "rare-race", "found-via-soak", "needs-live-repro", "needs-owner", "core-dump-inaccessible"]
related: ["bug-1696", "bug-1698"]
legacy: true
---

# bug-1699: ctrl-1 segfaulted (SIGSEGV, core dumped -- confirmed via bash's own accurate job-control message…

## Description
ctrl-1 segfaulted (SIGSEGV, core dumped -- confirmed via bash's own accurate job-control message, not a misread) within ~1.5s of a genuinely fresh boot, immediately after processing its first client 'ctrlstatus' probe request, during a fresh 10-round --chaotic run-ha-soak.sh --chaotic invocation (build.dbg, no ASAN) launched to verify bug-1696. Driver reported 'FAIL: no primary elected within ~90s' as a consequence (ctrl-1 never came back; ctrl-2 was subsequently Killed by cleanup).

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
NOT ROOT-CAUSED -- occurred exactly ONCE and could not be reproduced despite two dedicated repro attempts: (1) re-running the identical build.dbg chaotic soak under run-ha-soak.sh's built-in GDB_WRAP=ctrl-1 debug seam caught only a benign SIGPIPE (gdb's own default stop-on-SIGPIPE behavior firing on a normal peer-disconnect-during-TLS-write event that the real code already ignores via signal(SIGPIPE, SIG_IGN) at netio.c:1250 -- confirmed this was a gdb artifact, not the real crash, since production code never sees this signal); (2) a manual gdb session with `handle SIGPIPE nostop noprint pass` set before `run` (correctly letting SIGPIPE pass through), driving 15+ rapid ctrlstatus probes against a freshly-booted ctrl-1 via xce_test's REPL (mirroring the exact probe pattern run-ha-soak.sh's own wait-primary/discover_primary functions use), did NOT reproduce any crash. (3) A full, from-scratch 10-round --chaotic soak (33 jobs, 10 kill/elect cycles) run under a fresh ASAN rebuild (src/build-asan-1696repro) completed with a clean PASS (committed_divergence=0 committed_job_loss=0) and no SIGSEGV/ASAN report at all. This strongly suggests either a genuinely rare, narrow-window race (possibly specific to cold dynamic-linker/first-request timing that a debug or ASAN build's different memory layout/timing perturbs away) or an environment-specific fluke from that one specific boot -- NOT a deterministic, reliably-triggerable defect. No core file was accessible for postmortem gdb analysis (systemd-coredump is configured as the core_pattern handler but coredumpctl is not installed/reachable in this sandbox). FURTHER NOTE (2026-08-26): did not recur across the same 20+ reproduction attempts made while investigating bug-1698 (fresh boots, standby kill/restart cycles, load, exact user recipe replay) -- still zero occurrences since the single original sighting. FURTHER NOTE (2026-08-26, cont.): USER-SUPPLIED core dumps from the real deployment (4 core.xce_controller.*.zst files today) confirm this is a real, recurring heap corruption (glibc malloc_printerr: "unaligned tcache chunk detected", detected inside endpoint_get()/calloc for a 312-byte endpoint_t -- the abort site is downstream of the actual corruption, not the cause). Confirmed the crash DID occur at least once in this session too (attempt 7 of an earlier 8-attempt loop) but was MISSED by an earlier flawed stderr-grep detection method (setsid nohup ...  & disown fully detaches a process from the shells job table, so bash never prints a Aborted/Segmentation fault line for it -- a real methodology gap, not evidence of rarity). Redone with PROPER job-control-based exit-code detection: 15 attempts under a fresh ASAN build (build-asan-1696repro) = 0 reproductions; 15 more attempts with all 3 nodes wrapped in a properly-SIGPIPE-configured gdb session against build.dbg (where it previously reproduced) = 0 reproductions. Total 30 further attempts, zero catches. Hypothesis: ASANs different allocator/timing may suppress the race entirely (consistent with 0/15 under ASAN); build.dbg without a live gdb/gdb overhead may still need many more attempts or a specific concurrent-connection pattern not yet isolated. bug-1698s stuck-reconnect symptom and this heap corruption are plausibly the SAME root cause (both point at endpoint_t lifecycle -- corrupted/UAFd endpoint state could equally explain a connection that never completes ITS handshake). Next step if revisited: run MANY more attempts under build.dbg with proper exit-code detection (not ASAN, which appears to suppress it), or request the user run the real deployment under a debug build with core dumps enabled and share the NEXT core file for backtrace analysis the same way this session did.

## Fix
NOT ATTEMPTED -- cannot fix an unreproduced crash responsibly. If this recurs, capture a live gdb session (with SIGPIPE explicitly passed through, per this investigation's corrected GDB_WRAP-equivalent technique) or install/access a working coredumpctl-equivalent BEFORE the crashing process is cleaned up, to get a real backtrace with symbols. Also worth checking core_pattern/ulimit -c in the actual execution environment ahead of time so a future occurrence leaves a usable core file automatically.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1696, bug-1698