lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-drive-async): fix bug-1376 cold-start self-deadlock, flip peerctrl+snapshot flags ON

Browse source 
Root cause: peer_ctrl_msg_handler held ep->dispatch_inflight incremented
across the entire peer_ctrl_dispatch_to_pool call, including its inline
fallback (the common cold-start shape when ha->pool isn't wired yet). That
inline body can reach raft_cb_become_follower, which -- now that
cluster_init_phase2 ran synchronously earlier in the same call chain and
wired the pool -- tears down its own loser vote-ep (the very ep whose reply
triggered this call) via the deferred path (vote_ep_teardown_async), whose
endpoint_mark_tearing_down spins waiting for dispatch_inflight==0 on that
SAME ep. That count is held only by this exact call, on this exact thread,
and can't reach 0 until the spin itself returns -- a permanent self-deadlock.

Confirmed live via a gdb thread backtrace: added a GDB_WRAP debug seam to
run-ha-soak.sh (launches a named node directly under `gdb --batch` so it
becomes ptrace-legal as gdb's own direct child under yama scope 1, bypassing
a live-attach restriction), then SIGINT to gdb once the log went stale for a
bounded interrupt + full backtrace. The frame dump showed pool=0x0 at
peer_ctrl_dispatch_to_pool, and the same ep pointer in the nested
vote_ep_teardown_async -> endpoint_mark_tearing_down frame, spinning on
sched_yield().

Fix: moved the dispatch_inflight release into peer_ctrl_dispatch_to_pool
itself, at the exact right point per exit path -- released BEFORE calling
the inline body on every fallback path (pool==NULL, OOM, enqueue failure),
since io_worker is single-threaded and the inline path never enqueues
anything (no pool-worker race left to protect against once inline execution
is committed to); released only AFTER the non-blocking enqueue completes on
the success path, preserving the original protection for the genuinely
concurrent pool-worker case.

New unit test test_vep_self_teardown_does_not_deadlock uses a bounded
pthread_timedjoin_np so a regression fails loud instead of hanging the
suite. RED confirmed via a temporary manual revert (ETIMEDOUT) before
restoring the fix.

Live-verified with both flags ON: --force-snapshot soak (the exact repro
shape) completes without hanging across repeated runs, confirms real
snapshot installs, zero divergence, zero job loss; a 6-round --chaotic soak
also passed clean; full controller suite green throughout.

g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch_enabled
now default ON in production, closing raft-drive-async tasks 3.1 and 3.2.
g_drive_async_configpush_offload stays OFF -- a distinct, still-open hazard
in push_config_transport_only's endpoint handling (task 3.3, unrelated to
this bug).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014dZVSx3CdBjys9BB6hqpxP
This commit is contained in:
Rongsong Shen 2026-07-02 07:18:42 +00:00
parent 10f92a5d49
commit 1ce007814d
5 changed files with 272 additions and 70 deletions
Download patch file Download diff file Expand all files Collapse all files

75
.wolf/buglog.json
View file

@ -22813,15 +22813,14 @@
"timestamp": "2026-07-02T05:10:00.000Z",
"error_message": "raft-drive-async: cluster hangs silently (no crash, no ASAN report) during a --force-snapshot soak with g_drive_async_snapshot_offload + g_drive_async_peerctrl_dispatch_enabled both ON: a freshly-wiped empty node cold-starts, self-elects, then goes silent after 'peer_ctrl_dispatch_to_pool: vtask pool not wired -- running inline' is immediately followed by cluster_init_phase2 wiring the pool while raft_cb_become_follower's deferred-teardown vtasks (bug-1197's mechanism) are being enqueued -- the process never logs another line and is eventually killed by the test harness timeout.",
"file": "src/controller/ha.c",
"root_cause": "NOT YET ROOT-CAUSED. Hypothesis: a cold-start race between a fresh node's own self-triggered election (which can fire before cluster_init_phase2 finishes wiring ha->pool) and cluster_init_phase2 wiring the vtask pool concurrently -- raft_cb_become_follower's deferred vote-ep teardown (bug-1197/task 3.4a) enqueues VTASK_IO teardown vtasks bound to eps, and if the pool wiring or vtaskmgr worker startup is not yet fully complete at that exact moment, the enqueued teardown (or a subsequent dispatch) may never run, wedging the node. Only reproduces via a genuinely EMPTY node's cold start (--force-snapshot's deliberate wipe) -- a 6-round --chaotic soak with both flags ON passed clean because it restarts nodes with WAL intact, never exercising a truly empty cold start.",
"fix": "NOT YET FIXED. Mitigated by reverting g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch_enabled to their production-default OFF (commit f3b6e6a) -- production behavior unchanged, task 3.1/3.2/3.3 (raft-drive-async) stay open. Next step: a deterministic unit/integration test driving cluster_init_phase2 concurrently with an inbound vote reply on a freshly-constructed cluster_t, both flags ON, to reproduce the race without a full live soak.",
"root_cause": "peer_ctrl_msg_handler (ha.c) held ep->dispatch_inflight incremented across the ENTIRE peer_ctrl_dispatch_to_pool call, including its inline fallback (the common cold-start shape when ha->pool is not yet wired). That inline body can reach raft_cb_become_follower, which -- now that cluster_init_phase2 ran synchronously earlier in the SAME call chain and wired the pool -- tears down its own loser vote-ep (the very ep whose reply triggered this call) via the deferred path (vote_ep_teardown_async), whose endpoint_mark_tearing_down spins waiting for dispatch_inflight==0 on that SAME ep. That count is held only by this exact call, on this exact thread, and cannot reach 0 until the spin itself returns -- a permanent self-deadlock. Confirmed live via a gdb thread backtrace (GDB_WRAP debug seam added to run-ha-soak.sh, launching the target node directly under `gdb --batch` so it becomes ptrace-legal as gdb's own direct child, then SIGINT to gdb once the log went stale for a bounded interrupt + full backtrace): frame `peer_ctrl_dispatch_to_pool` showed pool=0x0 at dispatch time; the same ep pointer appeared in the nested vote_ep_teardown_async -> endpoint_mark_tearing_down frame, spinning on sched_yield().",
"fix": "Moved the ep->dispatch_inflight release into peer_ctrl_dispatch_to_pool itself, at the exact right point per exit path: released BEFORE calling the inline body on every fallback path (pool==NULL, OOM, enqueue failure) -- safe because io_worker is single-threaded and the inline path never enqueues anything, so there is no pool-worker race left to protect against once inline execution is committed to -- and only AFTER the (non-blocking) enqueue completes on the success path, preserving the original protection for the genuinely concurrent pool-worker case. New unit test test_vep_self_teardown_does_not_deadlock (test_ha_vep_teardown.c) uses a bounded-timeout pthread_timedjoin_np so a regression fails loud instead of hanging the suite; RED confirmed via a temporary manual revert (Expected 0 Was 110 == ETIMEDOUT) before restoring the fix. Live-verified with both g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch_enabled flipped ON: --force-snapshot soak (the exact repro shape) completes without hanging across repeated runs, confirmed real snapshot installs, committed_divergence=0, committed_job_loss=0; a 6-round --chaotic soak with both flags ON also passed clean; full controller suite green throughout. Both flags now default ON in production (raft-drive-async tasks 3.1/3.2 closed).",
"tags": [
"raft-drive-async",
"cold-start",
"vtask-pool",
"hang",
"deadlock-suspect",
"unresolved"
"self-deadlock",
"fixed"
],
"related_bugs": [
"bug-1197",
@ -22829,7 +22828,71 @@
"bug-1310"
],
"occurrences": 1,
"last_seen": "2026-07-02T05:10:00.000Z"
"last_seen": "2026-07-02T00:20:00.000Z"
},
{
"id": "bug-1377",
"timestamp": "2026-07-02T07:03:54.861Z",
"error_message": "Missing error handling in unknown",
"file": "src/controller/testsuite/test_ha_vep_teardown.c",
"root_cause": "Code path had no error handling \u2014 exceptions would propagate uncaught",
"fix": "Added try/catch block",
"tags": [
"auto-detected",
"error-handling",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-07-02T07:03:54.861Z"
},
{
"id": "bug-1378",
"timestamp": "2026-07-02T07:05:07.924Z",
"error_message": "Missing error handling in unknown",
"file": "src/controller/ha.c",
"root_cause": "Code path had no error handling \u2014 exceptions would propagate uncaught",
"fix": "Added try/catch block",
"tags": [
"auto-detected",
"error-handling",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-07-02T07:05:07.924Z"
},
{
"id": "bug-1379",
"timestamp": "2026-07-02T07:12:59.577Z",
"error_message": "Significant refactor of ",
"file": "src/controller/ha.c",
"root_cause": "23 lines replaced/restructured",
"fix": "Rewrote 30\u219242 lines (23 removed)",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-07-02T07:12:59.577Z"
},
{
"id": "bug-1380",
"timestamp": "2026-07-02T07:16:35.420Z",
"error_message": "Incorrect value in code",
"file": ".kiro/specs/raft-drive-async/tasks.md",
"root_cause": "Had `, differential-verified: this specific soak scena",
"fix": "Changed to `, differential-verified). (4) Re-ran `",
"tags": [
"auto-detected",
"wrong-value",
"md"
],
"related_bugs": [],
"occurrences": 2,
"last_seen": "2026-07-02T07:17:36.171Z"
}
]
}