forked from lei.chen/myxce
Rongsong Shen
1ce007814d
Root cause: peer_ctrl_msg_handler held ep->dispatch_inflight incremented across the entire peer_ctrl_dispatch_to_pool call, including its inline fallback (the common cold-start shape when ha->pool isn't wired yet). That inline body can reach raft_cb_become_follower, which -- now that cluster_init_phase2 ran synchronously earlier in the same call chain and wired the pool -- tears down its own loser vote-ep (the very ep whose reply triggered this call) via the deferred path (vote_ep_teardown_async), whose endpoint_mark_tearing_down spins waiting for dispatch_inflight==0 on that SAME ep. That count is held only by this exact call, on this exact thread, and can't reach 0 until the spin itself returns -- a permanent self-deadlock. Confirmed live via a gdb thread backtrace: added a GDB_WRAP debug seam to run-ha-soak.sh (launches a named node directly under `gdb --batch` so it becomes ptrace-legal as gdb's own direct child under yama scope 1, bypassing a live-attach restriction), then SIGINT to gdb once the log went stale for a bounded interrupt + full backtrace. The frame dump showed pool=0x0 at peer_ctrl_dispatch_to_pool, and the same ep pointer in the nested vote_ep_teardown_async -> endpoint_mark_tearing_down frame, spinning on sched_yield(). Fix: moved the dispatch_inflight release into peer_ctrl_dispatch_to_pool itself, at the exact right point per exit path -- released BEFORE calling the inline body on every fallback path (pool==NULL, OOM, enqueue failure), since io_worker is single-threaded and the inline path never enqueues anything (no pool-worker race left to protect against once inline execution is committed to); released only AFTER the non-blocking enqueue completes on the success path, preserving the original protection for the genuinely concurrent pool-worker case. New unit test test_vep_self_teardown_does_not_deadlock uses a bounded pthread_timedjoin_np so a regression fails loud instead of hanging the suite. RED confirmed via a temporary manual revert (ETIMEDOUT) before restoring the fix. Live-verified with both flags ON: --force-snapshot soak (the exact repro shape) completes without hanging across repeated runs, confirms real snapshot installs, zero divergence, zero job loss; a 6-round --chaotic soak also passed clean; full controller suite green throughout. g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch_enabled now default ON in production, closing raft-drive-async tasks 3.1 and 3.2. g_drive_async_configpush_offload stays OFF -- a distinct, still-open hazard in push_config_transport_only's endpoint handling (task 3.3, unrelated to this bug). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014dZVSx3CdBjys9BB6hqpxP |
||
|---|---|---|
| .. | ||
| include | ||
| testsuite | ||
| apiserver.c | ||
| CLAUDE.md | ||
| cluster.c | ||
| commit.c | ||
| config_repl.c | ||
| config_repl.h | ||
| ctrl_metrics.c | ||
| ctrlstatus.c | ||
| dataex.c | ||
| egress.c | ||
| ha.c | ||
| ha.h | ||
| ha_convergence.c | ||
| job.c | ||
| join.c | ||
| main.c | ||
| meson.build | ||
| misc.c | ||
| myapp.c | ||
| myconf.c | ||
| opget.c | ||
| opupd.c | ||
| persist.c | ||
| queue.c | ||
| repl.c | ||
| reply.c | ||
| resources.c | ||
| sched.c | ||
| slowlane.c | ||
| startup_validation.c | ||
| task.c | ||
| worker.c | ||