Rongsong Shen
426728bb98
Raft-drive-async's reconnect-offload burst (Task 2.1/2.2) exposed a latent race in io_worker's epoll dispatch loop: ha.c's on_standby_connected() evicts a stale duplicate peer channel via iomgr_unmonitor()+endpoint_unref(), but epoll_ctl(DEL) only stops FUTURE epoll_wait() calls -- it does not invalidate an event for that same endpoint already sitting in the CURRENT batch's events[] array. A later event in the same batch then dereferences the freed endpoint_t/SSL*, producing the ASAN SEGV observed deep inside OpenSSL's ssl3_read_bytes during raft-drive-async's round-9 soak. The original "concurrent SSL_do_handshake from two io_worker threads" hypothesis was wrong and is retracted -- iomgr_start() creates exactly one io_worker thread per iomgr_t. Fix: extract io_worker_dispatch_batch(), which takes endpoint_ref() on every endpoint in the batch before dispatching any of them and endpoint_unref()s each only after the whole batch completes -- the same lock-free ref/unref bracket already used for bug-1298 and INVOKE_MSG_HANDLER, just never applied to io_worker's own loop. Pure atomics, no locks, no blocking added to the hot path. Adds test_netio_batch_dispatch_uaf.c, which drives the real dispatch loop with a synthetic epoll_event[] batch (TDD RED confirmed via ASAN heap-use-after-free before the fix, GREEN after). Verified further with two independent clean 10-round live sequential HA soaks (33 and 36 jobs, zero loss) and full ASAN suite runs (xcelib_test 294/0, controller_unit_test 972/0). raft-drive-async's kiro-validate-impl re-run flips NO-GO to GO.
32 lines
5.7 KiB
JSON
32 lines
5.7 KiB
JSON
{
|
|
"feature_name": "raft-drive-async",
|
|
"created_at": "2026-06-25T11:37:13Z",
|
|
"updated_at": "2026-07-18T14:45:00Z",
|
|
"language": "en",
|
|
"phase": "implementation-complete",
|
|
"approvals": {
|
|
"requirements": {
|
|
"generated": true,
|
|
"approved": true
|
|
},
|
|
"design": {
|
|
"generated": true,
|
|
"approved": true
|
|
},
|
|
"tasks": {
|
|
"generated": true,
|
|
"approved": true
|
|
}
|
|
},
|
|
"ready_for_implementation": true,
|
|
"validate_impl": {
|
|
"decision": "GO",
|
|
"validated_at": "2026-07-18T14:45:00Z",
|
|
"note": "Re-validated after bug-1447 (the prior blocking finding) was root-caused and FIXED. \n\nMECHANICAL: Fresh ASAN build (nix develop + xce-build.sh lib, then controller) with the fix applied. TBD/TODO grep on this spec's owned files (slowlane.c/.h, vtaskmgr.c/.h, commit.c, netio.c/.h, ha.c, config_repl.c, cluster.c, config.h, ha.h, run-ha-soak.sh) found only the one pre-existing, already-flagged item (ha.c:1844, ha-term-safety-owned stub) -- zero new TBD/TODO/FIXME introduced by the fix. Secrets grep clean. Full test suite: libxce xcelib_test 294/0/0 (ASAN), controller_unit_test 972/0/0 (ASAN), zero AddressSanitizer/heap-use-after-free/heap-buffer-overflow/double-free/SEGV hits anywhere in the full controller test log.\n\nRUNTIME LIVENESS / SMOKE (the check that previously blocked): re-ran the canonical /xce-test-ha SEQUENTIAL 10-round soak TWICE on fresh ASAN binaries built with the fix, on real ext4 (not overlayfs, bug-924). Both fully clean: run 1 (/home/claude/tmp/xce-bug1447-verify-1) 10/10 rounds, 33 jobs, zero job loss, all 3 nodes led; run 2 (/home/claude/tmp/xce-bug1447-verify-2) 10/10 rounds, 36 jobs, zero job loss, all 3 nodes led including a forced-election leadership-coverage check on ctrl-3. An explicit grep of every node's log in both runs for bug-1447's exact signature (AddressSanitizer/DEADLYSIGNAL/SEGV/ssl3_read_bytes/handle_ssl_handshake) found zero matches. This is a concurrency/UAF bug class, so two independent clean 10-round soaks (66 rounds of kill/elect/reconnect churn combined across both verification runs) plus a deterministic unit regression test (below) are treated as the evidence bar, not a single pass.\n\nROOT CAUSE (bug-1447, now closed): NOT the originally-suspected 'two io_worker threads racing the same SSL*' -- there is exactly ONE io_worker thread per iomgr_t (iomgr_start creates one pthread), so that hypothesis was retracted. The real defect: io_worker's epoll dispatch loop processed every ready fd from ONE epoll_wait() batch in a plain for-loop with no reference held on any of them. ha.c's on_standby_connected (raft-drive-async Task 2.1/2.2's own reconnect-offload completion callback) evicts a stale duplicate outbound peer channel via iomgr_unmonitor()+endpoint_unref() when a fresher reconnect completes -- but epoll_ctl(EPOLL_CTL_DEL) only stops a FUTURE epoll_wait() from returning events for that fd; it does not invalidate an event for that SAME stale endpoint already sitting in the CURRENT batch's events[] array (very plausible during the reconnect burst this spec's own faster cadence produces right after a new-primary election). endpoint_unref() reaching refcnt 0 immediately freed the endpoint (and its SSL object); the loop's next iteration then read that freed memory in io_handler -- a heap-use-after-free that (when ep->ssl still looked like a plausible pointer) manifested as a deep SEGV inside OpenSSL's ssl3_read_bytes, exactly matching the original crash.\n\nFIX: src/lib/netio.c -- io_worker's per-batch dispatch was extracted into io_worker_dispatch_batch() (~line 639), which now takes an endpoint_ref() on every endpoint in the batch BEFORE dispatching any of them and endpoint_unref()s each one only AFTER the whole batch is processed -- the same endpoint_ref/endpoint_unref bracket idiom this codebase already uses for the vtask dispatch path (bug-1298) and INVOKE_MSG_HANDLER, applied here to the one dispatcher that didn't yet have it. Lock-free atomics only, no I/O under the bracket -- satisfies the async-io steering no-blocking-in-the-hot-path rule. New regression test src/lib/testsuite/test_netio_batch_dispatch_uaf.c drives the real batch-dispatch loop with a synthetic 2-event epoll batch reproducing the exact same-batch eviction; TDD-verified RED (ASAN heap-use-after-free at netio.c:587) before the fix, GREEN after. Full detail in .wolf/buglog.json bug-1447 (status FIXED).\n\nINTEGRATION / COVERAGE / DESIGN: netio.c is an explicitly-owned Modified File in this spec's File Structure Plan; the fix sits entirely inside io_worker's own dispatch loop (\"the scheduling discipline of drive-thread work\", squarely in this spec's Boundary Commitments -- This Spec Owns). No Revalidation Trigger fired (the fix does not change the slow-lane submit contract, does not make iomgr/delaymgr execute work synchronously, does not change per-connection ordering semantics, and does not touch socket_connect). No new files, no dependency-direction change (netio lib -> ha controller unchanged). Requirements coverage unchanged from the prior GO-track pass (this was a runtime defect found BY this spec's own verification task 4.2/4.1 soak gate, not a requirements gap). No tasks are _Blocked:_; all 17 tasks (1.1-4.3) remain [x].\n\nDistinguish from the separate, already-accepted chaotic-mode finding (task 4.2's notes): chaotic-mode's round-4 committed-prefix divergence remains the documented pre-existing bug-851/795 family, diff-confirmed against pre-Phase-3 binaries, correctly out of scope for this spec. bug-1447 was different (a hard crash in the fully-supported SEQUENTIAL lifecycle) and is now closed.",
|
|
"known_limitations": [],
|
|
"ownership": "LOCAL",
|
|
"upstream_spec": "N/A",
|
|
"blocking_bug": null,
|
|
"remediation": []
|
|
}
|
|
}
|