wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

feat(walcompare): add per-member checkpoint-floor awareness (raft-ha-core-safety 5.10a / ha-entity-snapshots 7.1)

Browse source 
walcompare previously had no concept of a per-member compaction floor:
committed_lsn only capped the HIGH end (skip LSNs a member hasn't
caught up to yet), but the LOW end unconditionally expected every
member to hold raw WAL from LSN 1. A member that legitimately caught
up via a full-checkpoint snapshot holds no raw WAL at or below its own
checkpoint LSN by design (that range was entity-restored, not
WAL-replayed) -- so it was always flagged MISSING/divergent there,
regardless of whether the restore was correct. This is the root cause
ha-entity-snapshots task 7.1 diagnosed (committed_divergence=11 on an
otherwise-healthy soak) and raft-ha-core-safety task 5.10a's residual.

Adds member_snapshot_floor(): each member's own snap/ sibling
directory (matching the deployed <data>/wal + <data>/snap layout) is
scanned for the highest installed snap-<lsn>.{capnp,bin} file. Any LSN
at or below that member's own floor is excluded from its comparison
the same way an LSN above its own committed_lsn already is -- never
uniform/shared across members, since compaction is a per-member fact.
wc_status() gains a new "compacted" token distinct from "MISSING" so
the machine-readable report can tell the two apart.

Two new regression tests prove: (1) the floor correctly suppresses the
compacted range from comparison, and (2) it does NOT mask a genuine
divergence within the range a member still claims to hold. Caught and
fixed one bug myself before landing: the floor comparison was
off-by-one (< instead of <=), which the first test run surfaced
immediately (Expected 0 Was 1).

test_walcompare 17/17, test_ha_core_safety 28/0, test_config_repl
66/0, test_waldump 7/0 (all no-regression). Soak --selftest 17/17.
This commit is contained in:
Rongsong Shen 2026-07-02 02:59:08 +00:00
parent 2806a96c37
commit f3c86a683a
3 changed files with 246 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

115
src/controller/testsuite/walcompare.c
View file

@ -1,8 +1,11 @@
#define _GNU_SOURCE
#include "walcompare.h"
#include <ctype.h>
#include <dirent.h>
#include <errno.h>
#include <inttypes.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@ -16,6 +19,16 @@
typedef struct {
uint64_t recovered_tip; /* highest CRC-intact LSN held (waldump tip) */
uint64_t committed_lsn; /* effective committed boundary for this member */
/* raft-ha-core-safety task 5.10a / ha-entity-snapshots task 7.1: the
* highest checkpoint LSN this member has installed (0 = never installed
* a snapshot -- expect raw WAL from LSN 1, the pre-existing behavior,
* unchanged). A member that caught up via a full-checkpoint snapshot
* legitimately holds NO raw WAL entries at or below its own floor_lsn --
* that range was restored via entity content, not WAL replay -- so it
* must never be flagged MISSING there. Read once per member from its own
* snap/ directory (scan_snapshot_floor); never uniform across members
* (unlike committed_lsn, which the caller may force to one shared value). */
uint64_t floor_lsn;
uint64_t cap; /* allocated index size (LSN range [0..cap-1]) */
uint64_t* term; /* raw on-disk term per LSN (0 = legacy / absent) */
uint32_t* pcrc; /* payload CRC per LSN */
@ -96,6 +109,86 @@ static void member_log_free(member_log_t* m)
m->cap = 0;
}
/* Scan one directory for `snap-<16-hex-digit-lsn>.{capnp,bin}` files (the
* on-disk naming write_snapshot()/legacy checkpoints use — persist.c) and
* return the highest embedded LSN found, or 0 if the directory doesn't exist
* or holds no matching file. A bare 16-hex-digit match is required (skips
* `.tmp` in-progress files and anything else) so this never mis-parses an
* unrelated entry as a checkpoint LSN. */
static uint64_t scan_snap_dir_floor(const char* dir)
{
DIR* d = opendir(dir);
if (!d)
return 0; /* no such dir (no snapshot ever installed here) — floor 0 */
uint64_t max_lsn = 0;
struct dirent* de;
while ((de = readdir(d)) != NULL) {
unsigned long long lsn = 0;
char ext[8];
/* "snap-XXXXXXXXXXXXXXXX.capnp" / ".bin" — %16llx would also accept
* fewer than 16 digits, so additionally check the literal width via
* strlen on the numeric run; simplest robust check: require the
* filename to be exactly "snap-" + 16 hex chars + "." + ext. */
size_t len = strlen(de->d_name);
if (len < 5 + 16 + 1 + 3 || strncmp(de->d_name, "snap-", 5) != 0)
continue;
int matched = sscanf(de->d_name + 5, "%16llx.%7s", &lsn, ext);
if (matched != 2)
continue;
if (strcmp(ext, "capnp") != 0 && strcmp(ext, "bin") != 0)
continue;
/* Confirm the hex run is EXACTLY 16 chars (reject a short-digit name
* like "snap-1.capnp" that sscanf's %16llx would still happily take). */
int hexlen = 0;
while (hexlen < 16 && isxdigit((unsigned char)de->d_name[5 + hexlen]))
hexlen++;
if (hexlen != 16 || de->d_name[5 + 16] != '.')
continue;
if ((uint64_t)lsn > max_lsn)
max_lsn = (uint64_t)lsn;
}
closedir(d);
return max_lsn;
}
/* Find this member's own checkpoint floor (raft-ha-core-safety task 5.10a /
* ha-entity-snapshots task 7.1). `src->path` may be either a "data" dir
* containing sibling `wal/`+`snap/` subdirs (the soak harness's convention:
* `$TOP/controller/ctrl-N/data`), or the WAL dir itself passed directly (seen
* in unit-test call sites, e.g. `persist_t.wal_path`) — in the latter case the
* sibling snap dir sits at the same parent level with `wal` replaced by
* `snap`. Try both; a nonexistent candidate just yields floor 0 (harmless). */
static uint64_t member_snapshot_floor(const char* path)
{
if (!path || !*path)
return 0;
char candidate[4160];
uint64_t floor1 = 0, floor2 = 0;
/* Candidate 1: <path>/snap (path is a "data" parent dir). */
snprintf(candidate, sizeof(candidate), "%s/snap", path);
floor1 = scan_snap_dir_floor(candidate);
/* Candidate 2: sibling of <path> with a trailing "/wal" (or "wal")
* component replaced by "/snap" (path is the WAL dir itself). */
size_t plen = strlen(path);
const char* base = strrchr(path, '/');
const char* leaf = base ? base + 1 : path;
if (strcmp(leaf, "wal") == 0) {
size_t prefix_len = (size_t)(leaf - path);
if (prefix_len + 4 < sizeof(candidate)) {
memcpy(candidate, path, prefix_len);
snprintf(candidate + prefix_len, sizeof(candidate) - prefix_len, "snap");
floor2 = scan_snap_dir_floor(candidate);
}
}
(void)plen;
return floor1 > floor2 ? floor1 : floor2;
}
/* Load one member's WAL into `*m`. Returns 0 or a negative errno. */
static int load_member(const walcompare_member_t* src, member_log_t* m, FILE* out)
{
@ -129,6 +222,10 @@ static int load_member(const walcompare_member_t* src, member_log_t* m, FILE* ou
/* An explicit committed_lsn wins; 0 means "derive from the shared
* boundary" (filled by the caller after all members are loaded). */
m->committed_lsn = src->committed_lsn;
/* Always PER-MEMBER, never uniform/shared (unlike committed_lsn) — a
* member's own checkpoint floor has nothing to do with any other
* member's state. */
m->floor_lsn = member_snapshot_floor(src->path);
return 0;
}
@ -146,6 +243,10 @@ static const char* wc_status(const member_log_t* m, uint64_t lsn)
{
if ((lsn < m->cap) && m->present[lsn])
return "present";
if (lsn <= m->floor_lsn)
return "compacted"; /* below this member's OWN checkpoint floor —
* legitimately absent (entity-restored, not
* WAL-held), never a divergence signal */
if (lsn <= m->committed_lsn)
return "MISSING"; /* a committed entry this member SHOULD hold (R4.2) */
return "beyond"; /* legitimately not held (above its committed prefix) */
@ -247,6 +348,9 @@ static int wc_compare_committed(const wc_cluster_t* cl,
for (int i = 0; i < cl->n; i++) {
if (lsn > cl->logs[i].committed_lsn)
continue; /* beyond member i's committed prefix */
if (lsn <= cl->logs[i].floor_lsn)
continue; /* below member i's OWN checkpoint floor — legitimately
* compacted (entity-restored), never a divergence */
have++;
const member_log_t* m = &cl->logs[i];
int holds = (lsn < m->cap) && m->present[lsn];
@ -276,8 +380,8 @@ static int wc_compare_committed(const wc_cluster_t* cl,
div->committed = 1;
for (int i = 0; i < cl->n && i < 16; i++) {
const member_log_t* m = &cl->logs[i];
int holds = (lsn <= m->committed_lsn) && (lsn < m->cap) &&
m->present[lsn];
int holds = (lsn <= m->committed_lsn) && (lsn > m->floor_lsn) &&
(lsn < m->cap) && m->present[lsn];
div->member_present[i] = holds ? 1 : 0;
div->member_term[i] = holds ? match_term(m->term[lsn]) : 0;
div->member_pcrc[i] = holds ? m->pcrc[lsn] : 0;
@ -296,6 +400,13 @@ static int wc_compare_committed(const wc_cluster_t* cl,
i, members[i].path, cl->logs[i].committed_lsn);
continue;
}
if (lsn <= cl->logs[i].floor_lsn) {
fprintf(out,
" member %d ('%s'): LSN below its own checkpoint floor"
" (floor=%" PRIu64 ") — legitimately compacted\n",
i, members[i].path, cl->logs[i].floor_lsn);
continue;
}
int holds = (lsn < cl->logs[i].cap) && cl->logs[i].present[lsn];
if (holds)
fprintf(out, " member %d ('%s'): term=%" PRIu64 " pcrc=%08x\n",