fix(raft-drive-async): fix bug-1376 cold-start self-deadlock, flip peerctrl+snapshot flags ON
Root cause: peer_ctrl_msg_handler held ep->dispatch_inflight incremented across the entire peer_ctrl_dispatch_to_pool call, including its inline fallback (the common cold-start shape when ha->pool isn't wired yet). That inline body can reach raft_cb_become_follower, which -- now that cluster_init_phase2 ran synchronously earlier in the same call chain and wired the pool -- tears down its own loser vote-ep (the very ep whose reply triggered this call) via the deferred path (vote_ep_teardown_async), whose endpoint_mark_tearing_down spins waiting for dispatch_inflight==0 on that SAME ep. That count is held only by this exact call, on this exact thread, and can't reach 0 until the spin itself returns -- a permanent self-deadlock. Confirmed live via a gdb thread backtrace: added a GDB_WRAP debug seam to run-ha-soak.sh (launches a named node directly under `gdb --batch` so it becomes ptrace-legal as gdb's own direct child under yama scope 1, bypassing a live-attach restriction), then SIGINT to gdb once the log went stale for a bounded interrupt + full backtrace. The frame dump showed pool=0x0 at peer_ctrl_dispatch_to_pool, and the same ep pointer in the nested vote_ep_teardown_async -> endpoint_mark_tearing_down frame, spinning on sched_yield(). Fix: moved the dispatch_inflight release into peer_ctrl_dispatch_to_pool itself, at the exact right point per exit path -- released BEFORE calling the inline body on every fallback path (pool==NULL, OOM, enqueue failure), since io_worker is single-threaded and the inline path never enqueues anything (no pool-worker race left to protect against once inline execution is committed to); released only AFTER the non-blocking enqueue completes on the success path, preserving the original protection for the genuinely concurrent pool-worker case. New unit test test_vep_self_teardown_does_not_deadlock uses a bounded pthread_timedjoin_np so a regression fails loud instead of hanging the suite. RED confirmed via a temporary manual revert (ETIMEDOUT) before restoring the fix. Live-verified with both flags ON: --force-snapshot soak (the exact repro shape) completes without hanging across repeated runs, confirms real snapshot installs, zero divergence, zero job loss; a 6-round --chaotic soak also passed clean; full controller suite green throughout. g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch_enabled now default ON in production, closing raft-drive-async tasks 3.1 and 3.2. g_drive_async_configpush_offload stays OFF -- a distinct, still-open hazard in push_config_transport_only's endpoint handling (task 3.3, unrelated to this bug). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014dZVSx3CdBjys9BB6hqpxP
This commit is contained in:
parent
10f92a5d49
commit
1ce007814d
5 changed files with 272 additions and 70 deletions
|
|
@ -186,6 +186,19 @@ start_node() {
|
|||
local extra_args=()
|
||||
[ -n "$MAX_WAL_RECORDS" ] && extra_args+=(--max-wal-records "$MAX_WAL_RECORDS")
|
||||
[ -n "$WAL_SEGMENT_BYTES" ] && extra_args+=(--wal-segment-bytes "$WAL_SEGMENT_BYTES")
|
||||
# bug-1376 debug seam: GDB_WRAP=<node-name> launches that node directly
|
||||
# under `gdb --batch` (gdb becomes its direct parent -> ptrace-legal under
|
||||
# yama scope 1) instead of via start.sh, so a hang can be interrupted
|
||||
# (SIGINT to gdb's own pid) for a live thread backtrace. Not for normal
|
||||
# soak use -- temporary investigation-only seam.
|
||||
if [ -n "${GDB_WRAP:-}" ] && [ "$GDB_WRAP" = "$1" ]; then
|
||||
gdb --batch -ex run -ex "thread apply all bt full" -ex "kill inferior 1" -ex quit \
|
||||
--args "$BIN" --myname "$1" --peers "$PEERS" \
|
||||
--workdir "$TOP/controller" --certdir "$TOP/certs" \
|
||||
--logfile "$TOP/logs/$1.log" --vcpus 4 -vvv "${extra_args[@]}" \
|
||||
>>"$TOP/logs/$1.gdbwrap" 2>&1 &
|
||||
return
|
||||
fi
|
||||
if [ ${#extra_args[@]} -gt 0 ]; then
|
||||
bash "$TOP/scripts/start.sh" "$1" -vvv "${extra_args[@]}" \
|
||||
>>"$TOP/logs/$1.stderr" 2>&1 &
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue