lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-drive-async): resolve snapshot chunk-drop (bug-1386) and chunk misordering (bug-1389)

Browse source 
ha_send_snapshot_to_standby ignored the network send's return value in its
chunk loop, silently truncating a transfer on a mid-stream error while still
logging "complete" (bug-1386). Fixing the silent drop exposed a second,
independent bug in the same pipeline: ha_handle_snapshot_chunk appended
received chunks in arrival order instead of at their wire-carried
chunk_offset, so out-of-order delivery across the vtask pool corrupted the
reassembled file even though every chunk's bytes were individually correct
(bug-1389). Fixed by checking every fallible call in the send loop and
switching the receiver to pwrite-at-offset with a count-based completion
tracker.

Adds snapverify/snapdiff xce_test commands (offline checkpoint-file
structural validator and byte-level comparator) used to diagnose bug-1389 and
kept as standing diagnostic tools for future snapshot-transfer triage.

g_drive_async_configpush_offload flipped back ON after both fixes, validated
via 6-round chaotic + 3/3 force-snapshot soak (raft-drive-async task 3.3).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
This commit is contained in:
Rongsong Shen 2026-07-03 02:40:12 +00:00
parent 02e46f6ba1
commit 5cb36794d2
14 changed files with 1553 additions and 43 deletions
Download patch file Download diff file Expand all files Collapse all files

46
src/controller/testsuite/snapverify.h Normal file
View file

@ -0,0 +1,46 @@
#pragma once
/*
* snapverify.h — read-only checkpoint-file structural inspector/validator
* (bug-1389 triage: OPCode_snapshotTransfer chunk-content corruption).
*
* Parses the on-disk checkpoint layout persist.c's load_snapshot() /
* load_snapshot_entities() use (design.md "Checkpoint File Layout") WITHOUT
* restoring any job/queue into a live cluster_t — a pure structural walk, so
* it needs no jobmgr/qmgr context and is safe to run standalone against a
* snap-*.capnp (locally generated) or snap-*.bin (received via
* ha_handle_snapshot_chunk, ha.c) file on disk.
*
* Strictly read-only: opens O_RDONLY, mutates nothing.
*/
#include <stdint.h>
#include <stdio.h>
#include <sys/types.h>
typedef struct {
uint64_t checkpoint_lsn;
int has_cfgref; /* config-ref tail present in the header */
int has_entity_section; /* fmt + job_count present past the header;
* 0 = legacy position-only checkpoint */
uint32_t format_version;
uint32_t job_count; /* as declared in the file (0 if legacy) */
uint32_t jobs_ok; /* job frames structurally walked before any failure */
uint32_t queue_count; /* as declared in the file (0 if legacy, or if the
* job-frame walk failed before reaching it) */
uint32_t queues_ok; /* queue frames structurally walked before any failure */
off_t file_size;
off_t consumed; /* bytes structurally accounted for */
int ok; /* 1 = well-formed, consumed == file_size */
off_t fail_offset; /* -1 if ok; else the byte offset of the field that
* failed the structural check */
char fail_reason[160];
} snapverify_report_t;
/* Parse+validate `path` (a snap-*.capnp or snap-*.bin checkpoint file).
* Prints a human-readable report to `out` (may be NULL to suppress) and
* fills `*rep` (may be NULL). Returns 0 if the file is structurally
* well-formed to EOF, or the same rc class persist_load_snapshot() would
* return for the equivalent defect (-EINVAL bad magic / unrecognized
* format_version, -EIO short read or a length field that doesn't fit the
* remaining bytes, -errno on open/fstat failure) — so a "persist_load_snapshot
* failed rc=..." log line correlates directly against this tool's verdict. */
int snapverify_path(const char* path, FILE* out, snapverify_report_t* rep);