Rongsong Shen
f3c86a683a
walcompare previously had no concept of a per-member compaction floor:
committed_lsn only capped the HIGH end (skip LSNs a member hasn't
caught up to yet), but the LOW end unconditionally expected every
member to hold raw WAL from LSN 1. A member that legitimately caught
up via a full-checkpoint snapshot holds no raw WAL at or below its own
checkpoint LSN by design (that range was entity-restored, not
WAL-replayed) -- so it was always flagged MISSING/divergent there,
regardless of whether the restore was correct. This is the root cause
ha-entity-snapshots task 7.1 diagnosed (committed_divergence=11 on an
otherwise-healthy soak) and raft-ha-core-safety task 5.10a's residual.
Adds member_snapshot_floor(): each member's own snap/ sibling
directory (matching the deployed <data>/wal + <data>/snap layout) is
scanned for the highest installed snap-<lsn>.{capnp,bin} file. Any LSN
at or below that member's own floor is excluded from its comparison
the same way an LSN above its own committed_lsn already is -- never
uniform/shared across members, since compaction is a per-member fact.
wc_status() gains a new "compacted" token distinct from "MISSING" so
the machine-readable report can tell the two apart.
Two new regression tests prove: (1) the floor correctly suppresses the
compacted range from comparison, and (2) it does NOT mask a genuine
divergence within the range a member still claims to hold. Caught and
fixed one bug myself before landing: the floor comparison was
off-by-one (< instead of <=), which the first test run surfaced
immediately (Expected 0 Was 1).
test_walcompare 17/17, test_ha_core_safety 28/0, test_config_repl
66/0, test_waldump 7/0 (all no-regression). Soak --selftest 17/17.
105 lines
5.4 KiB
C
105 lines
5.4 KiB
C
#pragma once
|
|
/*
|
|
* walcompare.h — cross-member WAL committed-prefix divergence detector
|
|
* (ha-term-safety R6.1/R6.2/R6.3).
|
|
*
|
|
* Reuses the canonical waldump.c WAL decoder (via waldump_path_visit) to index
|
|
* each member's log by `(lsn -> term, pcrc)`, then asserts that every member's
|
|
* COMMITTED prefix is identical by `(term, pcrc)` — not merely by LSN range.
|
|
*
|
|
* The decisive signal for bug-851: two members can report the same committed
|
|
* LSN over a contiguous, CRC-intact WAL yet hold DIFFERENT entries at the same
|
|
* LSN (same lsn, different term and/or payload). Comparing LSN ranges alone
|
|
* masks that; comparing `(term, pcrc)` per LSN surfaces it (R6.2).
|
|
*
|
|
* Strictly read-only — opens member WAL dirs O_RDONLY, mutates nothing. Safe to
|
|
* run against a live cluster's data dirs.
|
|
*/
|
|
#include <stdint.h>
|
|
#include <stdio.h>
|
|
|
|
/* A single member's WAL source and its committed boundary. */
|
|
typedef struct {
|
|
const char* path; /* WAL dir / data dir / seg-*.wal (waldump rules) */
|
|
uint64_t committed_lsn; /* highest committed LSN for this member; 0 =
|
|
* "derive" (use the shared boundary, see below) */
|
|
} walcompare_member_t;
|
|
|
|
/* Identity of the first divergent entry, filled on a WALCMP_DIVERGENT result. */
|
|
typedef struct {
|
|
uint64_t lsn; /* the diverging LSN within the committed prefix */
|
|
int n_members; /* members compared at this LSN */
|
|
uint64_t member_term[16]; /* per-member matched term at lsn */
|
|
uint32_t member_pcrc[16]; /* per-member payload CRC at lsn */
|
|
int member_present[16]; /* 1 if the member holds this LSN, else 0 */
|
|
int committed; /* 1 = divergence within the committed prefix
|
|
* (lsn <= committed boundary, a safety
|
|
* violation); 0 = uncommitted tail divergence */
|
|
} walcompare_divergence_t;
|
|
|
|
/* Result codes. */
|
|
enum {
|
|
WALCMP_OK = 0, /* all members' committed prefixes are identical */
|
|
WALCMP_DIVERGENT = 1, /* a committed-prefix divergence was found */
|
|
/* negative values are -errno (a load/usage error: missing WAL, bad args) */
|
|
};
|
|
|
|
/* Compare `n` members' committed WAL prefixes by `(lsn -> term, pcrc)`.
|
|
*
|
|
* Committed boundary: each member with `committed_lsn == 0` adopts the SHARED
|
|
* boundary = the minimum recovered tip (highest CRC-intact LSN) across all
|
|
* members, i.e. the largest prefix every member can be expected to hold. A
|
|
* member with an explicit `committed_lsn` uses that. An LSN in `[1..boundary]`
|
|
* is compared only against the members whose own boundary covers it; a member
|
|
* that is legitimately AHEAD (longer log) or behind the shared boundary is not
|
|
* flagged for the uncommitted tail it doesn't share (R6.1 is the committed
|
|
* prefix, not the full log).
|
|
*
|
|
* Per-member checkpoint floor (raft-ha-core-safety task 5.10a / ha-entity-
|
|
* snapshots task 7.1): a member that caught up via a full-checkpoint snapshot
|
|
* legitimately holds no raw WAL entries at or below its own installed
|
|
* checkpoint LSN — that range was restored via entity content, not WAL
|
|
* replay. Each member's floor is read independently from its own `snap/`
|
|
* sibling directory (0 if none installed, the unchanged pre-existing
|
|
* behavior); an LSN below a member's own floor is excluded from that
|
|
* member's comparison the same way an LSN above its committed_lsn already
|
|
* is — never flagged MISSING/divergent. This is a PER-MEMBER floor, never a
|
|
* shared/uniform value (unlike committed_lsn, which the caller may force to
|
|
* one shared value across all members).
|
|
*
|
|
* Legacy-zero term (R1.8) is mapped to WAL_LEGACY_TERM for matching, so two
|
|
* pre-feature entries are not spuriously divergent.
|
|
*
|
|
* On the FIRST divergent LSN, fills `*div` (when non-NULL) and prints a report
|
|
* to `out`, then returns WALCMP_DIVERGENT. Returns WALCMP_OK on identical
|
|
* committed prefixes, or a negative errno on a load/usage error. `out` may be
|
|
* NULL to suppress the report.
|
|
*/
|
|
int walcompare(const walcompare_member_t* members, int n, FILE* out,
|
|
walcompare_divergence_t* div);
|
|
|
|
/* Same comparison as walcompare(), but ALSO emits a machine-checkable,
|
|
* per-member report over the full log: one
|
|
*
|
|
* WALCMP member=<i> lsn=<L> term=<T> pcrc=<8hex> status=<present|MISSING|beyond> committed=<0|1>
|
|
*
|
|
* line per (member, LSN) for every LSN in [1..max-recovered-tip], where
|
|
* `committed=1` iff the LSN is within the committed boundary. Each divergence
|
|
* additionally emits a
|
|
*
|
|
* WALCMP_DIVERGENCE lsn=<L> class=<committed|uncommitted> compared=<c> present=<p>
|
|
*
|
|
* classification line. A `committed` divergence is fatal (verdict
|
|
* WALCMP_DIVERGENT, `div->committed = 1`); an `uncommitted`-tail divergence
|
|
* (above the committed boundary) is reported but NON-fatal — the verdict stays
|
|
* WALCMP_OK when the committed prefix is identical. Field order is fixed and
|
|
* the tokens are stable so the harness can grep them. `out` must be non-NULL.
|
|
*
|
|
* Backward-compatible: walcompare() keeps its original concise output (the
|
|
* `== walcompare ==` header, the human divergence block, and the final
|
|
* `walcompare: OK` verdict line) and additionally emits the machine
|
|
* classification at a committed divergence; only walcompare_report() prints the
|
|
* full per-LSN table and scans the uncommitted tail.
|
|
*/
|
|
int walcompare_report(const walcompare_member_t* members, int n, FILE* out,
|
|
walcompare_divergence_t* div);
|