lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(config-upgrade-snapshot): materialize upgrade snapshot on config-marker apply

Browse source 
Adds an explicit case WAL_CONFIG_MARKER to apply_committed_entry,
replacing its no-op treatment, that forces a checkpoint at exactly the
marker's own (term, lsn) via two new helpers:

  persist_checkpoint_write_locked(p, checkpoint_lsn) -- the checkpoint
  core (pin lsn, scan config ref, write_snapshot, trim, rotate) taking
  an explicit LSN and never acquiring apply_pump_lock itself
  persist_checkpoint_marker_triggered(p, lsn) -- shares the existing
  checkpoint_in_progress CAS guard, calls the above with the marker's
  own lsn

This fixes two hazards design.md's "call persist_checkpoint() directly"
text didn't account for: (1) apply_committed_entry's live-pump caller
(persist_apply_pump_run, used by both primary and standby online-apply)
already holds apply_pump_lock across the dispatch call -- a direct
persist_checkpoint() call would self-deadlock on that mutex; (2)
last_applied_lsn/replay position only advance AFTER apply_committed_entry
returns at both call sites, so pinning checkpoint_lsn from
last_applied_lsn inside the marker case would label the checkpoint with
the previous entry's LSN, not the marker's own -- fixed by threading the
marker's own lsn parameter through explicitly. A -EBUSY outcome (a
concurrent periodic checkpoint mid-flight) is logged at TRACE and treated
as expected/self-resolving, never retried. The new case has no effect on
entity interpretation/re-homing (Req 1.8).

New test_upgrade_snapshot_apply.c drives WAL_CONFIG_MARKER through the
REAL production call paths (persist_apply_pump_run and boot replay via
persist_recover), not a bare apply_committed_entry() call, so the fix is
actually proven against the lock-holding context rather than assumed.

Also fixes a genuine regression in test_config_repl.c surfaced by
WAL_CONFIG_MARKER apply having a real side effect for the first time
(bug-1466): an fsync-worker-thread race (t42_wait_applied polling helper,
mirrors test_apply_pump.c's tap_wait_applied) and one test call site
trusting an unreliable raft-derived LSN. Surfaces (but does not fix, out
of this spec's boundary) a pre-existing ha-term-safety/raft-log-replication
gap: persist_write_checkpoint_marker bypasses ha_primary_log_append,
breaking its documented current_lsn==log_len+1 invariant once a
checkpoint interleaves with primary appends -- log_len feeds real
RequestVote freshness comparisons, not advisory-only as first assumed;
severity corrected in bug-1466. Task 5 of 19.

Requirements: 1.1, 1.5, 1.8, 4.4, 5.2, 5.3
This commit is contained in:
Rongsong Shen 2026-07-24 16:28:57 +00:00
parent 00cd26a927
commit 1da26b043e
7 changed files with 938 additions and 23 deletions
Download patch file Download diff file Expand all files Collapse all files

189
src/controller/persist.c
View file

@ -2678,6 +2678,51 @@ static void schedule_checkpoint(persist_t* persist)
checkpoint_task_fn, persist);
}
/* ------------------------------------------------------------------ */
/* Checkpoint core (shared by persist_checkpoint and the marker-apply */
/* trigger, config-upgrade-snapshot task 5) */
/* ------------------------------------------------------------------ */
/* persist_checkpoint_write_locked: pin checkpoint_lsn to the EXPLICIT
* `checkpoint_lsn` given (never re-derived internally from
* last_applied_lsn), capture the committed-config reference, and serialize
* live state to a snapshot file.
*
* Caller MUST already hold apply_pump_lock (or otherwise guarantee no
* concurrent apply-pump walk can be racing this call, e.g. the
* single-threaded boot-replay window persist_recover() already documents
* elsewhere in this file) -- this function never locks or unlocks
* apply_pump_lock itself, so it is safe to call both from a context that
* already holds it (the live apply pump) and from a context where no lock
* is needed at all (boot replay).
*
* The explicit-lsn parameter exists because the two callers need two
* different values at two different moments:
* - persist_checkpoint() (periodic/explicit trigger): last_applied_lsn,
* read fresh by the caller immediately before this call.
* - the marker-apply trigger (apply_committed_entry's WAL_CONFIG_MARKER
* case): the marker's OWN lsn -- last_applied_lsn is NOT yet advanced
* to the marker's position at the moment that case runs (both live
* apply and replay advance it strictly AFTER dispatching this call),
* so reading last_applied_lsn from inside that case would mislabel the
* checkpoint with the marker's PREDECESSOR's lsn, not its own. */
static int persist_checkpoint_write_locked(persist_t* p,
uint64_t checkpoint_lsn)
{
p->checkpoint_lsn = checkpoint_lsn;
/* Capture the latest committed config-marker reference BEFORE trimming
* (ha-term-safety task 4.2 r1): trim_wal_segments (run by both callers,
* after this function returns) can unlink the segment holding a
* committed WAL_CONFIG_MARKER, so the derived committed config must be
* cached durably (into the sentinel) before that happens. */
scan_latest_committed_config_ref(p);
/* Serialize in-memory state to a snapshot file (P0: no live kvmgrs —
* writes sentinel header + cached config ref). */
return write_snapshot(p);
}
/* ------------------------------------------------------------------ */
/* Public: persist_checkpoint */
/* ------------------------------------------------------------------ */
@ -2714,18 +2759,8 @@ int persist_checkpoint(persist_t* p)
* the label ever claiming more than jobmgr actually holds. */
pthread_mutex_lock(&p->apply_pump_lock);
p->checkpoint_lsn = atomic_load_explicit(&p->last_applied_lsn,
memory_order_acquire);
/* 3a. Capture the latest committed config-marker reference BEFORE trimming
* (ha-term-safety task 4.2 r1): the trim below can unlink the segment
* holding a committed WAL_CONFIG_MARKER, so the derived committed config
* must be cached durably (into the sentinel) before that happens. */
scan_latest_committed_config_ref(p);
/* 3b. Serialize in-memory state to a snapshot file
* (P0: no live kvmgrs — writes sentinel header + cached config ref) */
rc = write_snapshot(p);
rc = persist_checkpoint_write_locked(
p, atomic_load_explicit(&p->last_applied_lsn, memory_order_acquire));
pthread_mutex_unlock(&p->apply_pump_lock);
@ -2743,6 +2778,61 @@ done:
return rc;
}
/* persist_checkpoint_marker_triggered (config-upgrade-snapshot task 5, Req
* 1.1, 1.5, 1.8, 4.4, 5.2, 5.3): force an immediate checkpoint at exactly
* `marker_lsn` -- the configuration-change marker's OWN lsn -- called from
* apply_committed_entry's WAL_CONFIG_MARKER case.
*
* Unlike persist_checkpoint(), this function NEVER locks apply_pump_lock:
* it relies entirely on the caller's own locking context, because the two
* production call sites of apply_committed_entry() differ in that context
* and this function must be correct under BOTH without deadlocking under
* either:
* - persist_apply_pump_run() (live online-apply, both primary and
* standby route through this) holds apply_pump_lock across its ENTIRE
* per-LSN dispatch loop, including the call into apply_committed_entry
* that reaches this function. Re-locking that same, non-recursive
* mutex here would self-deadlock -- so this function must NOT lock it;
* it is already held by the caller for the whole duration of this call.
* - replay_wal() (boot/restart replay, persist_recover()'s sole caller)
* does NOT hold apply_pump_lock during its per-entry dispatch loop --
* by design, since persist_recover() runs before the apply pump is
* ever kicked and nothing else can be racing it (see persist_recover's
* own doc comment on its brief, unrelated apply_pump_lock use for
* cache invalidation). No lock is needed here either, for the same
* reason.
* Both call sites are internal to persist.c and fully controlled, so this
* documented precondition (caller already serializes, one way or another)
* is safe to rely on rather than adding a redundant/ambiguous lock here.
*
* Shares checkpoint_in_progress -- the SAME single-in-flight CAS guard
* persist_checkpoint() uses -- so a concurrent WAL-threshold-triggered
* periodic checkpoint and this marker-triggered one can never both
* proceed at once (Req 5.3). Returns -EBUSY when that guard is already
* held by someone else; the caller (apply_committed_entry) treats that as
* an expected, self-resolving condition, not a failure (Req 4.4) -- the
* in-flight checkpoint (whichever triggered it) still completes and
* either it or a later checkpoint picks up this marker's configuration
* (every checkpoint embeds the currently active configuration, task 2.3). */
static int persist_checkpoint_marker_triggered(persist_t* p,
uint64_t marker_lsn)
{
if (atomic_exchange(&p->checkpoint_in_progress, 1) != 0)
return -EBUSY;
int rc = persist_write_checkpoint_marker(p);
if (rc == 0)
rc = persist_checkpoint_write_locked(p, marker_lsn);
if (rc == 0) {
trim_wal_segments(p);
rotate_snapshots(p);
}
atomic_store(&p->checkpoint_in_progress, 0);
return rc;
}
/* ------------------------------------------------------------------ */
/* Recovery internals */
/* ------------------------------------------------------------------ */
@ -3641,10 +3731,79 @@ int apply_committed_entry(persist_t* p, uint32_t wal_type, uint64_t lsn,
return 0;
}
case WAL_CONFIG_MARKER: {
/* config-upgrade-snapshot task 5 (Req 1.1, 1.5, 1.8): force an
* immediate, deterministic checkpoint at exactly this marker's own
* (term, lsn) boundary, materializing the upgrade snapshot the
* moment every member applies the marker -- primary online-apply,
* standby online-apply, and boot/restart replay alike, since this
* one unified apply seam is exactly what all three route through
* (design.md "Architecture" / "Key simplification found during
* design": no new leader-specific "resume" logic is needed either,
* since a newly-elected leader satisfies this the same way, by
* continuing to apply its own committed log).
*
* Deliberately does NOT touch entity interpretation/re-homing
* (lost_and_found etc.) -- Req 1.8 requires apply and activation to
* stay separate steps; this case's only effect is materialization,
* identical in kind to any other checkpoint.
*
* persist_checkpoint_marker_triggered() (not a bare
* persist_checkpoint() call) is required here to avoid two real
* hazards a naive direct call would hit:
* 1. Lock reentrancy: persist_apply_pump_run() (the live pump's
* call site for online-apply, both primary and standby) holds
* apply_pump_lock across this entire dispatch call;
* persist_checkpoint()'s own body re-acquires that same,
* non-recursive mutex, which would self-deadlock.
* persist_checkpoint_marker_triggered() never locks
* apply_pump_lock itself -- see its own doc comment for why
* that is safe under both production call sites.
* 2. Stale checkpoint_lsn: both live-apply and replay advance
* last_applied_lsn AFTER this dispatch call returns, never
* before (persist_apply_pump_run / replay_wal), so reading
* last_applied_lsn from inside this case would capture the
* PRECEDING entry's lsn, not the marker's own, mislabeling the
* checkpoint. `lsn` (this function's own parameter) is
* already the marker's exact position, so it is passed
* through explicitly instead of re-derived. */
int rc = persist_checkpoint_marker_triggered(p, lsn);
if (rc == -EBUSY) {
/* Expected, self-resolving (Req 4.4/5.3): a concurrent
* WAL-threshold-triggered or earlier marker-triggered
* checkpoint already holds the shared single-in-flight guard
* (checkpoint_in_progress). That in-flight checkpoint
* completes on its own; this marker's own configuration is
* still picked up by it or a later checkpoint (every
* checkpoint embeds the currently active configuration, task
* 2.3) -- not an error, no retry loop here. */
TRACE("apply_committed_entry: WAL_CONFIG_MARKER lsn=%" PRIu64
" checkpoint already in progress -- skipping, "
"self-resolving", lsn);
return 0;
}
if (rc != 0) {
/* A genuine checkpoint failure (I/O error, etc.) -- propagate
* so the caller's existing halt-and-retry-from-same-position
* handling engages (persist_apply_pump_run / replay_wal),
* exactly mirroring every other negative-return case in this
* dispatch. last_applied_lsn is NOT advanced past this marker,
* so the next kick/retry re-attempts the checkpoint fresh
* (Req 5.2 crash-safety: resume materialization from the
* marker). */
ERROR("apply_committed_entry: WAL_CONFIG_MARKER lsn=%" PRIu64
" forced checkpoint failed (rc=%d)", lsn, rc);
return rc;
}
return 0;
}
default:
/* WAL_NOOP / WAL_CONFIG_MARKER / WAL_CHECKPOINT / unknown — no
* entity effect at this seam, no hook (unchanged from
* apply_wal_entry's existing treatment of these types). */
/* WAL_NOOP / WAL_CHECKPOINT / unknown — no entity effect at this
* seam, no hook (unchanged from apply_wal_entry's existing
* treatment of these types). WAL_CONFIG_MARKER has its own explicit
* case above (config-upgrade-snapshot task 5) and no longer falls
* through to here. */
return 0;
}
}