fix(raft-drive-async): fix bug-1376 cold-start self-deadlock, flip peerctrl+snapshot flags ON

Root cause: peer_ctrl_msg_handler held ep->dispatch_inflight incremented
across the entire peer_ctrl_dispatch_to_pool call, including its inline
fallback (the common cold-start shape when ha->pool isn't wired yet). That
inline body can reach raft_cb_become_follower, which -- now that
cluster_init_phase2 ran synchronously earlier in the same call chain and
wired the pool -- tears down its own loser vote-ep (the very ep whose reply
triggered this call) via the deferred path (vote_ep_teardown_async), whose
endpoint_mark_tearing_down spins waiting for dispatch_inflight==0 on that
SAME ep. That count is held only by this exact call, on this exact thread,
and can't reach 0 until the spin itself returns -- a permanent self-deadlock.

Confirmed live via a gdb thread backtrace: added a GDB_WRAP debug seam to
run-ha-soak.sh (launches a named node directly under `gdb --batch` so it
becomes ptrace-legal as gdb's own direct child under yama scope 1, bypassing
a live-attach restriction), then SIGINT to gdb once the log went stale for a
bounded interrupt + full backtrace. The frame dump showed pool=0x0 at
peer_ctrl_dispatch_to_pool, and the same ep pointer in the nested
vote_ep_teardown_async -> endpoint_mark_tearing_down frame, spinning on
sched_yield().

Fix: moved the dispatch_inflight release into peer_ctrl_dispatch_to_pool
itself, at the exact right point per exit path -- released BEFORE calling
the inline body on every fallback path (pool==NULL, OOM, enqueue failure),
since io_worker is single-threaded and the inline path never enqueues
anything (no pool-worker race left to protect against once inline execution
is committed to); released only AFTER the non-blocking enqueue completes on
the success path, preserving the original protection for the genuinely
concurrent pool-worker case.

New unit test test_vep_self_teardown_does_not_deadlock uses a bounded
pthread_timedjoin_np so a regression fails loud instead of hanging the
suite. RED confirmed via a temporary manual revert (ETIMEDOUT) before
restoring the fix.

Live-verified with both flags ON: --force-snapshot soak (the exact repro
shape) completes without hanging across repeated runs, confirms real
snapshot installs, zero divergence, zero job loss; a 6-round --chaotic soak
also passed clean; full controller suite green throughout.

g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch_enabled
now default ON in production, closing raft-drive-async tasks 3.1 and 3.2.
g_drive_async_configpush_offload stays OFF -- a distinct, still-open hazard
in push_config_transport_only's endpoint handling (task 3.3, unrelated to
this bug).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014dZVSx3CdBjys9BB6hqpxP
This commit is contained in:
Rongsong Shen 2026-07-02 07:18:42 +00:00
parent 10f92a5d49
commit 1ce007814d
5 changed files with 272 additions and 70 deletions

View file

@ -186,6 +186,19 @@ start_node() {
local extra_args=()
[ -n "$MAX_WAL_RECORDS" ] && extra_args+=(--max-wal-records "$MAX_WAL_RECORDS")
[ -n "$WAL_SEGMENT_BYTES" ] && extra_args+=(--wal-segment-bytes "$WAL_SEGMENT_BYTES")
# bug-1376 debug seam: GDB_WRAP=<node-name> launches that node directly
# under `gdb --batch` (gdb becomes its direct parent -> ptrace-legal under
# yama scope 1) instead of via start.sh, so a hang can be interrupted
# (SIGINT to gdb's own pid) for a live thread backtrace. Not for normal
# soak use -- temporary investigation-only seam.
if [ -n "${GDB_WRAP:-}" ] && [ "$GDB_WRAP" = "$1" ]; then
gdb --batch -ex run -ex "thread apply all bt full" -ex "kill inferior 1" -ex quit \
--args "$BIN" --myname "$1" --peers "$PEERS" \
--workdir "$TOP/controller" --certdir "$TOP/certs" \
--logfile "$TOP/logs/$1.log" --vcpus 4 -vvv "${extra_args[@]}" \
>>"$TOP/logs/$1.gdbwrap" 2>&1 &
return
fi
if [ ${#extra_args[@]} -gt 0 ]; then
bash "$TOP/scripts/start.sh" "$1" -vvv "${extra_args[@]}" \
>>"$TOP/logs/$1.stderr" 2>&1 &