wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

feat(raft-apply-decoupling): post-replay apply-correctness verification (task 5.1)

Browse source 
Add apply_verdict_t (APPLY_CLEAN/APPLY_DIVERGENT) computed alongside
the existing structural (CRC) recover_verdict_t. Dual-check design:
an entity-count check (WAL_JOB_SUBMIT entries scanned vs live jobmgr
growth, genuinely independent of apply_committed_entry()'s return
code) and a position check against a new last_applied_lsn field
(forward-looking infrastructure for task 6's live pump).

Fixes a real bug found during review: the position check's
expectation was computed from the physical WAL-scan tip
(current_lsn - 1), which advances past checkpoint markers and
unknown entry types that never reach apply_committed_entry(). A
crash between persist_checkpoint()'s marker-fsync step and its
later snapshot-persist step leaves a trailing checkpoint marker on
next boot, which falsely reported APPLY_DIVERGENT on a perfectly
clean recovery -- reachable today, not gated behind a future task.
Fixed by reading last_applied_lsn directly, which only advances for
entries actually dispatched through the apply seam.

Known accepted gap: entity-count check covers jobs only, not
workers (which can genuinely be created via WAL_WORKER_CTRL) --
documented for future extension.
This commit is contained in:
Rongsong Shen 2026-07-04 11:32:21 +00:00
parent 665b29f5b0
commit c005d40741
6 changed files with 800 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

99
src/include/persist.h
View file

@ -87,6 +87,52 @@ typedef struct {
uint64_t cluster_hwm_lsn; /* cluster committed high-water mark, for logging */
} recover_policy_inputs_t;
/* ============================================================
* Post-replay apply-correctness verdict (raft-apply-decoupling task 5.1,
* design.md "Restart Replay & Verification", Req 8.1/8.2)
*
* DISTINCT from recover_verdict_t above, and NOT a replacement for it:
* RECOVER_CLEAN/RECOVER_TRUNCATED is a structural verdict over the WAL's
* bytes (CRC chain intact or not). apply_verdict_t is a SEMANTIC verdict
* over the result of replaying those bytes through apply_committed_entry() —
* it proves the materialized state (e.g. the live job table) actually
* matches a lightweight expectation derived from the committed log tip, not
* merely that the bytes were readable. This catches a class of hidden apply
* defect (e.g. a dispatch call that reports success without the entity
* actually landing in the live table) that a byte-level CRC check cannot
* see. Both verdicts are reported side by side on every restart (Req 8.1);
* neither one implies the other.
* ============================================================ */
typedef enum {
APPLY_CLEAN = 0, /* materialized state matches the log-derived
* expectation (Req 8.2). */
APPLY_DIVERGENT = 1, /* materialized state does NOT match the log-derived
* expectation (Req 8.3) — a hidden apply defect.
* Routing this into the existing non-promotable
* fence mechanism is raft-apply-decoupling task 5.2;
* this type only classifies, it does not fence. */
} apply_verdict_t;
/* Lightweight post-replay expectation, derived purely from scanning the
* committed log during replay (task 5.1, Req 8.1) — never from trusting
* apply_committed_entry()'s own return code, so the comparison is a genuine
* second, independent check on the materialized result.
*
* expected_last_applied_lsn — the highest LSN of any logical (non-
* checkpoint, non-unknown-type) entry replay actually dispatched via
* apply_committed_entry() in this replay pass: the "committed log tip"
* position component of the expectation.
* expected_new_job_count — the number of WAL_JOB_SUBMIT entries replay
* dispatched in this pass. Every such entry is, by construction, above
* the loaded snapshot's checkpoint_lsn (entries at/below it are skipped
* before ever reaching dispatch — see replay_wal), so each one names a
* job that must now exist in the live job table that did NOT exist
* before this replay pass began. */
typedef struct {
uint64_t expected_last_applied_lsn;
uint64_t expected_new_job_count;
} apply_expectation_t;
/* ============================================================
* Persistence configuration (Part 2.6)
* ============================================================ */
@ -278,6 +324,27 @@ struct persist_s {
* threads. Defaults to false before recovery runs. */
_Atomic bool recovery_unrecoverable;
/* Post-replay apply-correctness verdict + cursor (raft-apply-decoupling
* task 5.1, design.md "Restart Replay & Verification", Req 8.1/8.2).
* last_applied_lsn is the replay-derived LOGICAL apply cursor — the
* highest LSN actually dispatched via apply_committed_entry() — seeded
* from checkpoint_lsn before replay begins and advanced by replay_wal
* per entry; it is DISTINCT from current_lsn (the physical WAL position
* owned by the physical layer, bug-778) even though the two coincide on
* a clean replay. Never persisted — recomputed fresh every
* persist_recover() (design.md State Management), so it cannot itself
* drift from disk state. The live apply pump (task 6.1, not yet built)
* takes over its advancement once replay completes. apply_verdict is the
* sticky APPLY_CLEAN/APPLY_DIVERGENT result of comparing this cursor (and
* the live job table's growth) against the log-derived expectation;
* defaults to APPLY_CLEAN before recovery runs, mirroring
* recovery_verdict's default-clean convention. Written single-threaded
* during recovery; read by persist_apply_verdict/persist_last_applied_lsn
* thereafter (task 5.2 will additionally consume apply_verdict to route a
* DIVERGENT result into the existing loud-fence mechanism). */
_Atomic uint64_t last_applied_lsn;
apply_verdict_t apply_verdict;
/* Paths (derived from workdir + name via persist_set_workdir) */
char wal_path[4096];
char snap_path[4096];
@ -607,6 +674,38 @@ persist_recover_loud_fence_if_unrecoverable(persist_t* p,
bool persist_recovery_unrecoverable(persist_t* p);
void persist_clear_recovery_unrecoverable(persist_t* p);
/* persist_verify_apply_correctness (raft-apply-decoupling task 5.1, Req
* 8.1/8.2). Pure comparison — mirrors persist_recover_decision's separation
* of decision from action: no logging, no fence side effects, no mutation of
* `p` beyond the read of last_applied_lsn. Compares `exp` (derived by
* scanning the committed log during replay — see apply_expectation_t) against
* the actual materialized state: p's replay-derived apply cursor, and — when
* p->cluster is non-NULL — the live job table's growth since
* `jobmgr_count_before_replay` (the job count captured just before replay
* began). Returns APPLY_CLEAN on a match. Returns APPLY_DIVERGENT when either
* the apply cursor or the live job count disagrees with `exp`.
*
* Safe to call with p->cluster == NULL (a bare unit-test persist_t, or any
* caller with nothing to dispatch into): the job-count half of the check is
* skipped — only the position check applies — mirroring
* apply_committed_entry()'s own "nothing to dispatch into" convention. Safe
* on NULL p or NULL exp (returns APPLY_CLEAN — nothing to verify). */
apply_verdict_t persist_verify_apply_correctness(
persist_t* p, const apply_expectation_t* exp,
uint64_t jobmgr_count_before_replay);
/* Accessor for the sticky verdict computed by the last persist_recover()
* (task 5.1). APPLY_CLEAN before recovery has run (fresh store default,
* mirrors persist_recovery_verdict's RECOVER_CLEAN default). Safe on NULL p. */
apply_verdict_t persist_apply_verdict(persist_t* p);
/* Accessor for the replay-derived logical apply cursor (task 5.1,
* design.md "Restart Replay & Verification" state model) — see the
* last_applied_lsn field doc comment for its precise semantics and how it
* differs from current_lsn/persist_durable_floor. Safe on NULL p (returns
* 0). Lock-free atomic read. */
uint64_t persist_last_applied_lsn(persist_t* p);
/* Register a durable-advance wake callback (ha-commit-durability task 2.2).
* The group-commit flusher calls fn(ctx) after it advances durable_lsn and
* after releasing p->lock, so a consumer (the commit engine) can re-evaluate