fix(controller): CtrlHelloRequest.key Data field crashed HA leader election (bug-1835)
CtrlHelloRequest.key was declared `Data` in controller.capnp. This project's capnp-c backend silently no-ops encoding of Data-typed fields without zero-initializing the caller's local struct, leaving an uninitialized capn_ptr that write_CtrlHelloRequest unconditionally wrote to the wire -- SIGSEGV in capn_setp/copy_ptr. Hit live: ctrl-1 won its Raft election and crashed calling ha_send_ctrlhello_to_all_standbys(). Same bug class as bug-1504/bug-1505, both already fixed by switching the field from Data to List(UInt8); bug-1505's own writeup had flagged CtrlHelloRequest.key as a known-but-unfixed instance. Applied the identical fix here: Data -> List(UInt8) + keyLen companion in the schema, uint8_t*/key_len in typemap.h. The field is unused in production (always NULL), so no consumer code changed. Added a round-trip regression test covering both the NULL-key production shape and real key content. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
1e43345b2e
commit
cd9ecb4159
7 changed files with 197 additions and 21 deletions
|
|
@ -24742,6 +24742,40 @@
|
|||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-20T10:15:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1835",
|
||||
"timestamp": "2026-09-21T03:47:10Z",
|
||||
"error_message": "Live 3-node HA cluster crash: xce_controller SIGSEGV in __memcpy_avx_unaligned_erms, called from capnp's copy_ptr/capn_setp, called from write_CtrlHelloRequest (lib/controller.capnp.c:22821) via encode_CtrlHelloRequest_ptr (lib/controller.capnp.c:22935), via xce_encode/xce_request, via ha_send_ctrlhello_to_all_standbys (controller/cluster.c:2432), via raft_cb_become_leader (controller/ha.c:2113), via raft_on_vote_reply -> vote_reply_dispatch -> peer_ctrl_msg_handler_inline -> peer_ctrl_vtask_entry, on the vcpu thread handling the winning vote reply. Core confirmed built from commit 1e43345b (xce v0.0.1-1e43345b, 2026-09-21T03:10:39Z); ctrl-1 had just won its Raft election (term 2, votes_granted=2) and crashed while broadcasting CtrlHello to its 2 standbys.",
|
||||
"file": "src/schema/controller.capnp",
|
||||
"root_cause": "CtrlHelloRequest.key was declared `Data` in controller.capnp -- the exact same latent bug class as bug-1504 (SnapshotData.internalState) and bug-1505 (WalCatchupEntry.payload), and explicitly named as a known-but-unfixed instance of it in bug-1505's own writeup ('Known/suspected other Data fields with the same latent gap: ... CtrlHelloRequest.key'). This project's capnp-c backend silently no-ops the encoding of any Data-typed field inside the generated encode_XXX() composite helper (see encode_CtrlHelloRequest's ' /* key key */' stub, lib/controller.capnp.c) without ever zero-initializing the caller's local `struct CtrlHelloRequest d;` first (encode_CtrlHelloRequest_ptr, lib/controller.capnp.c:22926-22938) -- so d.key.p is left as genuinely uninitialized stack garbage. write_CtrlHelloRequest (lib/controller.capnp.c:22821) then unconditionally calls capn_setp(p.p, 1, s->key.p) with that garbage capn_ptr, and capn_setp's copy_ptr dereferences the garbage segment/data pointer, SIGSEGV'ing in memcpy. ha_send_ctrlhello_to_all_standbys (controller/cluster.c:2419-2424) and ha.c:9495 both always set ctrl_hello_req_t.key = NULL (the field has never had a real producer anywhere in the codebase -- confirmed via project-wide grep, no call site ever sets it to non-NULL nor reads it back), so this is genuine undefined behavior whose manifestation depends entirely on whatever garbage bytes happen to occupy that vtask thread's stack slot at each call -- exactly the same 'lucky vs unlucky stack layout' nondeterminism documented for bug-1504/1505, which is why this had not crashed before despite the gap being known since 2026-08-07. Also confirmed the C struct itself compounded the risk: ctrl_hello_req_s.key (typemap.h) was typed `xce_data_t*` -- the local-Data-struct-shadowing wrapper type from bug-1461's own root cause -- rather than a real byte-blob type, even though it was never actually used as one.",
|
||||
"fix": "FIXED (2026-09-21), same pattern as bug-1504/bug-1505: changed CtrlHelloRequest.key from `Data` to `List(UInt8)` in src/schema/controller.capnp (which this capnp-c backend DOES correctly auto-generate real, non-no-op encode/decode/free for -- proven precedent: FetchFileReply.content/bytesRead, SnapshotData.internalState, WalCatchupEntry.payload), with a new keyLen (@4, UInt32) companion field as its maplistcount. Updated typemap.h's ctrl_hello_req_s.key from `xce_data_t*` to `uint8_t*` and added a new `key_len` (uint32_t) field. No consumer-code changes were needed: project-wide grep confirmed .key is never set to non-NULL nor read anywhere in production (ha_send_ctrlhello_to_all_standbys and ha.c's hello builder both always pass NULL), so the fix is purely structural -- it eliminates the uninitialized-capn_ptr class of UB for this field rather than papering over one occurrence. Added regression test test_ctrl_hello_request_key_round_trip (controller/testsuite/test_e2e_capnp.c), covering both the actual production shape (key==NULL/key_len==0, the exact shape that used to SIGSEGV -- now proven to encode/decode cleanly via the real generic codec) and a real non-NULL key payload (proves content genuinely round-trips now, which the old no-op'd Data codec never did). A deterministic RED test reproducing the SIGSEGV itself was not attempted -- by its own nature as reads of uninitialized stack memory, this bug class is not ASAN-catchable and its manifestation is stack-layout-dependent (bug-1504's own writeup makes the identical observation); the fix instead structurally eliminates the uninitialized-read entirely rather than chasing a flaky repro. Verified: full controller_unit_test suite (5/5 meson test binaries, 0 failures, including the new round-trip test and the pre-existing test_ctrl_hello_req_t_layout/test_ha_ctrlhello.c suite) via a clean ASAN rebuild (`xce-build.sh controller asan`) that also regenerated controller.capnp.c/.h from the updated schema with no codegen errors.",
|
||||
"tags": [
|
||||
"controller",
|
||||
"ha",
|
||||
"capnp-schema",
|
||||
"codegen-limitation",
|
||||
"sigsegv",
|
||||
"uninitialized-memory",
|
||||
"controller.capnp",
|
||||
"ctrl-hello",
|
||||
"raft",
|
||||
"leader-election",
|
||||
"systemic",
|
||||
"user-reported",
|
||||
"live-crash",
|
||||
"related-to-bug-1504",
|
||||
"related-to-bug-1505",
|
||||
"related-to-bug-1461",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1504",
|
||||
"bug-1505",
|
||||
"bug-1461"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-21T03:47:10Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue