---
id: "bug-1835"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "crash"
timestamp: "2026-09-21T03:47:10Z"
last_seen: "2026-09-21T03:47:10Z"
occurrences: 1
files: ["src/schema/controller.capnp"]
tags: ["controller", "ha", "capnp-schema", "codegen-limitation", "sigsegv", "uninitialized-memory", "controller.capnp", "ctrl-hello", "raft", "leader-election", "systemic", "user-reported", "live-crash", "related-to-bug-1504", "related-to-bug-1505", "related-to-bug-1461"]
related: ["bug-1504", "bug-1505", "bug-1461"]
legacy: true
gitea_issue: 64
gitea_synced_status: "fixed"
---

# bug-1835: Live 3-node HA cluster crash: xce_controller SIGSEGV in __memcpy_avx_unaligned_erms, called from…

## Description
Live 3-node HA cluster crash: xce_controller SIGSEGV in __memcpy_avx_unaligned_erms, called from capnp's copy_ptr/capn_setp, called from write_CtrlHelloRequest (lib/controller.capnp.c:22821) via encode_CtrlHelloRequest_ptr (lib/controller.capnp.c:22935), via xce_encode/xce_request, via ha_send_ctrlhello_to_all_standbys (controller/cluster.c:2432), via raft_cb_become_leader (controller/ha.c:2113), via raft_on_vote_reply -> vote_reply_dispatch -> peer_ctrl_msg_handler_inline -> peer_ctrl_vtask_entry, on the vcpu thread handling the winning vote reply. Core confirmed built from commit 1e43345b (xce v0.0.1-1e43345b, 2026-09-21T03:10:39Z); ctrl-1 had just won its Raft election (term 2, votes_granted=2) and crashed while broadcasting CtrlHello to its 2 standbys.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
CtrlHelloRequest.key was declared `Data` in controller.capnp -- the exact same latent bug class as bug-1504 (SnapshotData.internalState) and bug-1505 (WalCatchupEntry.payload), and explicitly named as a known-but-unfixed instance of it in bug-1505's own writeup ('Known/suspected other Data fields with the same latent gap: ... CtrlHelloRequest.key'). This project's capnp-c backend silently no-ops the encoding of any Data-typed field inside the generated encode_XXX() composite helper (see encode_CtrlHelloRequest's ' /* key key */' stub, lib/controller.capnp.c) without ever zero-initializing the caller's local `struct CtrlHelloRequest d;` first (encode_CtrlHelloRequest_ptr, lib/controller.capnp.c:22926-22938) -- so d.key.p is left as genuinely uninitialized stack garbage. write_CtrlHelloRequest (lib/controller.capnp.c:22821) then unconditionally calls capn_setp(p.p, 1, s->key.p) with that garbage capn_ptr, and capn_setp's copy_ptr dereferences the garbage segment/data pointer, SIGSEGV'ing in memcpy. ha_send_ctrlhello_to_all_standbys (controller/cluster.c:2419-2424) and ha.c:9495 both always set ctrl_hello_req_t.key = NULL (the field has never had a real producer anywhere in the codebase -- confirmed via project-wide grep, no call site ever sets it to non-NULL nor reads it back), so this is genuine undefined behavior whose manifestation depends entirely on whatever garbage bytes happen to occupy that vtask thread's stack slot at each call -- exactly the same 'lucky vs unlucky stack layout' nondeterminism documented for bug-1504/1505, which is why this had not crashed before despite the gap being known since 2026-08-07. Also confirmed the C struct itself compounded the risk: ctrl_hello_req_s.key (typemap.h) was typed `xce_data_t*` -- the local-Data-struct-shadowing wrapper type from bug-1461's own root cause -- rather than a real byte-blob type, even though it was never actually used as one.

## Fix
FIXED (2026-09-21), same pattern as bug-1504/bug-1505: changed CtrlHelloRequest.key from `Data` to `List(UInt8)` in src/schema/controller.capnp (which this capnp-c backend DOES correctly auto-generate real, non-no-op encode/decode/free for -- proven precedent: FetchFileReply.content/bytesRead, SnapshotData.internalState, WalCatchupEntry.payload), with a new keyLen (@4, UInt32) companion field as its maplistcount. Updated typemap.h's ctrl_hello_req_s.key from `xce_data_t*` to `uint8_t*` and added a new `key_len` (uint32_t) field. No consumer-code changes were needed: project-wide grep confirmed .key is never set to non-NULL nor read anywhere in production (ha_send_ctrlhello_to_all_standbys and ha.c's hello builder both always pass NULL), so the fix is purely structural -- it eliminates the uninitialized-capn_ptr class of UB for this field rather than papering over one occurrence. Added regression test test_ctrl_hello_request_key_round_trip (controller/testsuite/test_e2e_capnp.c), covering both the actual production shape (key==NULL/key_len==0, the exact shape that used to SIGSEGV -- now proven to encode/decode cleanly via the real generic codec) and a real non-NULL key payload (proves content genuinely round-trips now, which the old no-op'd Data codec never did). A deterministic RED test reproducing the SIGSEGV itself was not attempted -- by its own nature as reads of uninitialized stack memory, this bug class is not ASAN-catchable and its manifestation is stack-layout-dependent (bug-1504's own writeup makes the identical observation); the fix instead structurally eliminates the uninitialized-read entirely rather than chasing a flaky repro. Verified: full controller_unit_test suite (5/5 meson test binaries, 0 failures, including the new round-trip test and the pre-existing test_ctrl_hello_req_t_layout/test_ha_ctrlhello.c suite) via a clean ASAN rebuild (`xce-build.sh controller asan`) that also regenerated controller.capnp.c/.h from the updated schema with no codegen errors.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1504, bug-1505, bug-1461
- Gitea: #64