checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions

View file

@ -0,0 +1,42 @@
#pragma once
/*
* waldump.h — read-only WAL segment inspector for failover/durability triage.
*
* Decodes the on-disk WAL (`wal_entry_hdr_t` + payload, see wal_types.h),
* lists every physical entry, decodes JOB_SUBMIT jobids via the Cap'n Proto
* Job schema, and — the decisive signal for the committed-job-loss bug —
* flags LSN gaps (a missing prefix, or any non-contiguous hole) and reports
* the CRC-chain "recovered tip" a real recovery would stop at.
*
* Strictly read-only: opens segments O_RDONLY, applies no entries, mutates no
* persist_t state. Safe to run against a live node's data dir.
*/
#include <stdint.h>
#include <stdio.h>
/* Aggregate result of scanning one path (a seg file, a wal/ dir, or a node
* data dir). All counts are over the physically-present, CRC-valid entries. */
typedef struct {
uint64_t first_lsn; /* lowest LSN seen (0 = no entries) */
uint64_t last_lsn; /* highest LSN seen */
uint64_t recovered_tip; /* highest LSN reachable via an intact CRC chain */
uint32_t n_entries; /* total entries scanned */
uint32_t n_submits; /* WAL_JOB_SUBMIT entries */
uint32_t n_gaps; /* LSN discontinuities (missing prefix counts as 1) */
uint32_t n_segments; /* segment files scanned */
uint32_t n_dup_submits; /* JOB_SUBMITs with same jobid as the immediately-
* preceding entry at an adjacent LSN (bug-845) */
uint32_t n_dup_identical; /* of n_dup_submits, those whose payload is
* byte-identical to the prior entry */
int crc_broken; /* nonzero if a CRC/chain break truncated a segment */
} waldump_stats_t;
/* Dump every WAL entry under `path` to `out` (human-readable) and fill
* `*stats` when non-NULL. `path` may be:
* - a single seg-*.wal file,
* - a directory holding seg-*.wal files (a wal/ dir), or
* - a node/data dir whose wal/ subdir (data/wal or wal) holds the segments.
* Returns 0 on success, or a negative errno on a hard I/O error (e.g. the
* path does not exist / no segments found). Gaps and CRC breaks are reported
* through `out`/`stats`, NOT via the return code. */
int waldump_path(const char* path, FILE* out, waldump_stats_t* stats);