lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-apply-decoupling): bug-1419 layer 2 — checkpoint_lsn mislabeled from WAL tip instead of applied cursor

Browse source 
persist_checkpoint() stamped checkpoint_lsn from p->current_lsn (physical
WAL append tip) instead of p->last_applied_lsn (the true "jobmgr reflects
up through here" cursor), uncoordinated with the async apply pump
introduced by raft-apply-decoupling. write_snapshot() serializes live
jobmgr state, so a checkpoint taken while apply lags the WAL tip mislabels
the snapshot: on restart last_applied_lsn is re-seeded from checkpoint_lsn
and replay resumes strictly above it, silently skipping committed-but-not-
yet-applied entries jobmgr never received. This is the actual mechanism
behind kill-mid-apply producing a permanent APPLY_DIVERGENT fence —
apply_committed_entry() is deterministic/idempotent and write_snapshot()
is crash-atomic, so the bug was a mislabeled watermark, not a structural
divergence requiring a safety/liveness tradeoff.

Fix stamps checkpoint_lsn from last_applied_lsn and holds apply_pump_lock
across the label capture + write_snapshot() so the label and the jobmgr
state it describes are captured atomically w.r.t. the live apply pump.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
This commit is contained in:
Rongsong Shen 2026-07-08 00:05:19 +00:00
parent fe4352fb36
commit 0eb105431b
4 changed files with 292 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

28
src/controller/persist.c
View file

@ -2258,10 +2258,27 @@ int persist_checkpoint(persist_t* p)
if (rc != 0)
goto done;
/* 2. Record checkpoint_lsn = current position after the marker */
pthread_mutex_lock(&p->lock);
p->checkpoint_lsn = p->current_lsn;
pthread_mutex_unlock(&p->lock);
/* 2. Record checkpoint_lsn = the APPLIED cursor (last_applied_lsn), not
* the physical WAL append tip (current_lsn). write_snapshot() below
* serializes live jobmgr state, which only reflects entries the
* apply pump has actually dispatched -- NOT every entry durably
* appended/committed, since apply is decoupled from commit
* (raft-apply-decoupling) and can lag behind current_lsn under load.
* Stamping current_lsn here mislabels the snapshot: on restart,
* last_applied_lsn is re-seeded FROM checkpoint_lsn and replay
* resumes strictly above it, so any committed-but-not-yet-applied
* entries between the true apply cursor and current_lsn at
* checkpoint time are silently skipped forever, even though jobmgr
* never actually contained them (persist.h last_applied_lsn doc).
* Held under apply_pump_lock together with 3a/3b so the label and
* the jobmgr walk it describes are captured atomically w.r.t. the
* live apply pump -- the pump can only ever move last_applied_lsn
* and jobmgr forward, never backward, so freezing it here rules out
* the label ever claiming more than jobmgr actually holds. */
pthread_mutex_lock(&p->apply_pump_lock);
p->checkpoint_lsn = atomic_load_explicit(&p->last_applied_lsn,
memory_order_acquire);
/* 3a. Capture the latest committed config-marker reference BEFORE trimming
* (ha-term-safety task 4.2 r1): the trim below can unlink the segment
@ -2272,6 +2289,9 @@ int persist_checkpoint(persist_t* p)
/* 3b. Serialize in-memory state to a snapshot file
* (P0: no live kvmgrs — writes sentinel header + cached config ref) */
rc = write_snapshot(p);
pthread_mutex_unlock(&p->apply_pump_lock);
if (rc != 0)
goto done;