lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-drive-async): close snapshot-lane borrowed-endpoint hazard + catch-up checkpoint-marker bug

Browse source 
Both fixes are real and independently verified; the drive-async flags
(g_drive_async_snapshot_offload, g_drive_async_peerctrl_dispatch_enabled)
stay at their production-default OFF -- see the tasks.md session note for
why the flip itself doesn't land in this commit.

1. Snapshot-lane borrowed-endpoint hazard (task 3.2, bug-1197 family):
   ha_snapshot_send_dispatch now takes endpoint_ref(ep) before submitting to
   the slow lane, released on every non-lane terminal path and by
   snapshot_lane_fn on completion. Reuses the general bug-1298 endpoint
   refcount system (standby-ep teardown already routes through
   endpoint_unref via cluster.c's endpoint_killer) -- a concurrent RDHUP
   teardown can now only drop the owner's reference while a transfer is in
   flight, never free the ep out from under it. New test:
   test_snapshot_offload_holds_endpoint_ref_across_transfer (RED->GREEN).

2. Catch-up streamer WAL_CHECKPOINT bug (found while verifying #1 via a live
   soak, pre-existing, unrelated to the drive-async flags): ha_stream_wal_
   to_standby had no filter for WAL_CHECKPOINT markers -- local-only
   bookkeeping that still consumes a real LSN slot via persist_write_
   checkpoint_marker's own current_lsn++. It shipped one to a catching-up
   standby, which correctly refused it (persist_wal_write_replicated
   rejects type==WAL_CHECKPOINT) and got stuck retrying the same LSN
   forever. Fixed by skipping WAL_CHECKPOINT entirely in the streamer loop,
   mirroring the existing replay_wal/term-index precedent of treating
   checkpoint-marker LSNs as invisible gaps. New test:
   test_wal_stream_skips_checkpoint_marker (RED->GREEN).

Attempted flipping both drive-async flags ON after landing these fixes (the
"full fix, defer prod flip" decision's prerequisite -- 4.1+4.2 are green).
A 6-round --chaotic soak passed clean, but a --force-snapshot soak (a
genuinely empty node cold-start rejoin, the one scenario chaotic mode never
exercises) surfaced a THIRD, not-yet-root-caused issue: the freshly-wiped
node's own self-election appears to race cluster_init_phase2's vtask-pool
wiring, then the process hangs silently (no crash, no ASAN report) until
killed. Reverted both flags to OFF rather than ship an unproven state --
production behavior is unchanged by this commit. Full controller suite
green and both soaks (--force-snapshot, 6-round --chaotic) pass clean with
flags OFF, confirming no regression from the two logic fixes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014dZVSx3CdBjys9BB6hqpxP
This commit is contained in:
Rongsong Shen 2026-07-02 04:54:17 +00:00
parent 1439e2655e
commit f3b6e6ae3b
5 changed files with 315 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

19
src/controller/config_repl.c
View file

@ -2186,6 +2186,25 @@ int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
if (hdr.lsn == 0 && hdr.type == 0 && hdr.payload_len == 0) break;
if (hdr.lsn > committed_lsn) break;
/* WAL_CHECKPOINT is a local-only marker (persist_write_checkpoint_
* marker's own doc comment: "replay_wal SKIPS checkpoint markers...
* the recovered index never holds a checkpoint LSN; the live index
* must match") -- it consumes a real LSN slot in THIS node's local
* current_lsn sequence but is never a logical Raft log entry and is
* never sent on the live broadcast path either
* (persist_wal_write_replicated rejects type==WAL_CHECKPOINT with
* -EINVAL). Skip it entirely -- do not build/send a catchup entry
* and do not advance prev_streamed_lsn/prev_streamed_term, so the
* NEXT real entry's AppendEntries prev tuple chains to the last
* REAL entry before this gap, exactly matching what a standby that
* never records a checkpoint marker in its own log expects.
* Without this, a standby receiving one gets stuck retrying the
* same LSN forever (the receive-side guard correctly refuses it). */
if (hdr.type == WAL_CHECKPOINT) {
off += (uint32_t)(sizeof(hdr) + hdr.payload_len);
continue;
}
if (hdr.lsn > standby_lsn) {
wal_catchup_entry_t *entry = calloc(1, sizeof(*entry));
if (!entry) {