lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-drive-async): resolve snapshot chunk-drop (bug-1386) and chunk misordering (bug-1389)

Browse source 
ha_send_snapshot_to_standby ignored the network send's return value in its
chunk loop, silently truncating a transfer on a mid-stream error while still
logging "complete" (bug-1386). Fixing the silent drop exposed a second,
independent bug in the same pipeline: ha_handle_snapshot_chunk appended
received chunks in arrival order instead of at their wire-carried
chunk_offset, so out-of-order delivery across the vtask pool corrupted the
reassembled file even though every chunk's bytes were individually correct
(bug-1389). Fixed by checking every fallible call in the send loop and
switching the receiver to pwrite-at-offset with a count-based completion
tracker.

Adds snapverify/snapdiff xce_test commands (offline checkpoint-file
structural validator and byte-level comparator) used to diagnose bug-1389 and
kept as standing diagnostic tools for future snapshot-transfer triage.

g_drive_async_configpush_offload flipped back ON after both fixes, validated
via 6-round chaotic + 3/3 force-snapshot soak (raft-drive-async task 3.3).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
This commit is contained in:
Rongsong Shen 2026-07-03 02:40:12 +00:00
parent 02e46f6ba1
commit 5cb36794d2
14 changed files with 1553 additions and 43 deletions
Download patch file Download diff file Expand all files Collapse all files

43
src/controller/testsuite/snapdiff.h Normal file
View file

@ -0,0 +1,43 @@
#pragma once
/*
* snapdiff.h — byte-level comparator for two checkpoint files
* (bug-1389 triage: OPCode_snapshotTransfer chunk-content corruption).
*
* Compares a locally-generated source checkpoint (snap-*.capnp) against a
* received one (snap-*.bin, ha_handle_snapshot_chunk) byte-for-byte, and
* reports the first divergent offset plus whether it lands exactly on an
* OPCode_snapshotTransfer chunk boundary (SNAPSHOT_CHUNK_SIZE, config_repl.c)
* — the decisive signal separating "chunk transport corrupted this specific
* chunk" from "the two files were never going to match" (different content
* entirely, unrelated to chunking).
*
* Strictly read-only: opens both files O_RDONLY, mutates nothing.
*/
#include <stdio.h>
#include <sys/types.h>
/* Mirrors config_repl.c's SNAPSHOT_CHUNK_SIZE (the OPCode_snapshotTransfer
* wire chunk size) — kept in sync manually; both are compile-time constants
* for the same wire protocol, not expected to change independently. */
#define SNAPDIFF_CHUNK_SIZE 65536
typedef struct {
off_t size1;
off_t size2;
int size_mismatch; /* 1 if size1 != size2 (diff_offset unset in that case) */
int identical; /* 1 if same size and byte-for-byte identical */
off_t diff_offset; /* -1 if identical or size_mismatch; else first
* differing byte offset */
int at_chunk_boundary; /* 1 if diff_offset is the first byte of a
* SNAPDIFF_CHUNK_SIZE-aligned chunk, i.e.
* diff_offset % SNAPDIFF_CHUNK_SIZE == 0 */
off_t chunk_index; /* diff_offset / SNAPDIFF_CHUNK_SIZE, valid only
* when diff_offset >= 0 */
} snapdiff_report_t;
/* Compare `path1` (source of truth) against `path2` (candidate, e.g. a
* received copy). Prints a human-readable report to `out` (may be NULL) and
* fills `*rep` (may be NULL). Returns 0 if identical, 1 if they differ (see
* `*rep` for detail), or a negative errno on open/fstat failure. */
int snapdiff_paths(const char* path1, const char* path2, FILE* out,
snapdiff_report_t* rep);