wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(ha): close permanent standby catch-up wedge after a WAL_CHECKPOINT boundary (bug-1475)

Browse source 
A restarted standby could get permanently stuck unable to catch up,
with the primary re-streaming the identical doomed AppendEntries range
every heartbeat with zero progress -- discovered via a live 3-node
config-upgrade-snapshot soak. Liveness-only; no safety invariant
(Election Safety, Leader Completeness, State Machine Safety, Commit
Durability) was ever violated.

Three-layer root cause, all now closed:

1. ha_stream_wal_to_standby (config_repl.c) used to skip advancing its
   prev_streamed_lsn/prev_streamed_term tracker across WAL_CHECKPOINT
   entries (a local-only, never-replicated marker that still consumes
   a real LSN slot). Now advances across them too, while still never
   shipping a catchup entry for the checkpoint itself.

2. ha_apply_term_entry_via_library (ha.c) now JIT-resyncs
   ha->raft.log_len from persist_durable_floor() before consuming an
   AE, to correct for persist_write_checkpoint_marker advancing
   current_lsn without going through raft_log_append/record_entry_term
   (pre-existing gap, bug-1466, not itself fixed here).

3. The actual root cause, found via real instrumentation after layers
   1-2 alone still left a regression test red: record_entry_term's
   array-growth realloc zero-filled new slots, silently reclassifying
   a checkpoint's deliberately-never-recorded gap slot from "absent"
   to "present, term=0/WAL_LEGACY_TERM" the instant any later LSN got
   recorded -- a time-dependent read where a permanent fact was
   assumed. Fixed with a TERM_INDEX_UNSET (UINT64_MAX) sentinel that
   growth now fills gap slots with instead of zero, so entry_term_locked
   can distinguish "never explicitly written" from "explicitly
   recorded as legacy term 0" regardless of later array growth.

The same determinism fix exposed a second, independent instance of the
identical wedge via the snapshot-install catch-up path instead of
WAL-streaming catch-up: persist_reconcile_checkpoint_term recorded a
wire checkpoint_term of 0 as a raw, explicit write (bypassing the
growth-fill sentinel entirely), so a snapshot-installed standby read
the checkpoint LSN back as WAL_LEGACY_TERM(1) forever while the leader
now correctly read its own checkpoint LSN as absent(0) -- the same
permanent logOk mismatch, a different code path. Fixed by substituting
TERM_INDEX_UNSET for a wire value of exactly 0 before recording; a
wire 0 is unambiguous (persist_entry_term's own contract never returns
raw 0 for a present entry, so 0 can only mean genuine absence).

Independently reviewed across three rounds: sentinel mechanism
verified via real RED->GREEN revert-and-rebuild; a documentation
claim that config_repl.c/ha.c were "comment-only" corrected; the
snapshot-path gap independently traced, confirmed real via its own
RED->GREEN cycle, and fixed; two stale doc comments (persist.h's
public contract, ha.c's call-site rationale) still describing the
disproven pre-fix legacy-0 convention corrected. Suite: 1028 -> 1035
tests, 0 failures.
This commit is contained in:
Rongsong Shen 2026-07-28 04:48:53 +00:00
parent 7d31fce27a
commit ce8df99037
5 changed files with 650 additions and 45 deletions
Download patch file Download diff file Expand all files Collapse all files

469
src/controller/testsuite/test_config_repl.c
View file

@ -4476,17 +4476,32 @@ static int capture_all_entries_wal_stream_hook(endpoint_t *ep, xce_request_t *re
* checkpoint LSN; the live index must match"), never a logical Raft log
* entry, and never sent on the live broadcast path either
* (persist_wal_write_replicated rejects type==WAL_CHECKPOINT with -EINVAL).
* A standby receiving one gets stuck retrying the same LSN forever.
*
* Fixture: LSN [1..5], LSN 3 is a checkpoint marker (consumes a real LSN slot,
* exactly like a live checkpoint firing mid-range). Streaming standby_lsn=0,
* committed_lsn=5 must emit exactly 4 real entries (1,2,4,5), never lsn=3, and
* the entry after the gap (lsn=4) must chain prev_log_index=2 (the last REAL
* entry before the gap), not 3.
* committed_lsn=5 must emit exactly 4 real entries (1,2,4,5), never lsn=3.
*
* RED before the fix: the streamer builds/sends a catchup_entry for every
* hdr.type in range with no filter, so lsn=3/type=WAL_CHECKPOINT is included
* and lsn=4's prev_log_index is 3.
* bug-1475 (UPDATED, supersedes the original assertion): the entry after the
* gap (lsn=4) must now chain prev_log_index=3 -- the checkpoint's OWN lsn --
* NOT 2 (the last real entry before it). The original assertion here (prev_
* log_index=2, "not the skipped marker's lsn=3") encoded the OLD, DEFECTIVE
* design: a receiving standby that has itself already applied the same
* committed WAL_CONFIG_MARKER independently writes its OWN local checkpoint
* at the identical LSN (persist_write_checkpoint_marker runs deterministically
* on every member that applies the marker's commit, not just the sender), so
* the checkpoint's LSN is a real, HELD position on both sides -- not a gap to
* skip past. Never advancing the tracker across it left every subsequent
* prev_log_index permanently one below the receiver's true log length,
* producing a permanent post-checkpoint catch-up stall (ae_reject_logmismatch
* forever, zero progress every heartbeat) -- root-caused and fixed as part of
* bug-1475; see ha_apply_term_entry_via_library's bug-1475 fix (ha.c) for the
* matching receive-side co-tracking half of this fix, and
* test_bug1475_catchup_survives_midstream_checkpoint below for the full
* end-to-end (send+receive) regression proof.
*
* RED before the ORIGINAL bug (pre-existing, already fixed): the streamer
* built/sent a catchup_entry for every hdr.type with no filter at all, so
* lsn=3/type=WAL_CHECKPOINT itself was shipped as a real entry.
*/
static void test_wal_stream_skips_checkpoint_marker(void)
{
@ -4519,15 +4534,20 @@ static void test_wal_stream_skips_checkpoint_marker(void)
"no streamed entry may carry entry_type==WAL_CHECKPOINT");
}
/* Entries in order must be lsn 1,2,4,5 with lsn=4's prev_log_index=2
* (the last real entry before the checkpoint gap, not the marker's lsn 3). */
/* Entries in order must be lsn 1,2,4,5. */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(1, g_cap_ckpt_lsn[0], "entry 0 must be lsn=1");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(2, g_cap_ckpt_lsn[1], "entry 1 must be lsn=2");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(4, g_cap_ckpt_lsn[2], "entry 2 must be lsn=4 (lsn=3 skipped)");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(5, g_cap_ckpt_lsn[3], "entry 3 must be lsn=5");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(2, g_cap_ckpt_prev_idx[2],
"lsn=4 must chain prev_log_index=2 (the last real entry before the "
"checkpoint gap), not the skipped marker's lsn=3");
/* bug-1475: lsn=4 must chain prev_log_index=3 -- the checkpoint's OWN lsn,
* which is a real HELD position on any receiver that independently
* applied the same committed marker -- NOT 2 (the stale, pre-fix value
* that permanently wedged catch-up one entry later; see the updated
* fixture doc comment above and test_bug1475_catchup_survives_
* midstream_checkpoint for the full send+receive proof). */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(3, g_cap_ckpt_prev_idx[2],
"bug-1475: lsn=4 must chain prev_log_index=3 (the checkpoint's own "
"lsn, a real held position on both sides), not the stale lsn=2");
xce_wal_stream_request_hook = NULL;
@ -4537,6 +4557,216 @@ static void test_wal_stream_skips_checkpoint_marker(void)
printf("test_wal_stream_skips_checkpoint_marker: PASS\n");
}
/* ══════════════════════════════════════════════════════════════════════════
* bug-1475 — END-TO-END regression: a standby must NOT permanently stall
* catching up across a mid-stream WAL_CHECKPOINT.
*
* This drives all THREE parts of the fix together through the REAL production
* send+receive path — not just the sender's prev_log_index computation
* (already pinned above) in isolation:
* - SEND side (config_repl.c, ha_stream_wal_to_standby): the tracker now
* advances across a skipped WAL_CHECKPOINT instead of freezing one entry
* behind.
* - RECEIVE side (ha.c, ha_apply_term_entry_via_library): JIT-resyncs
* ha->raft.log_len from persist_durable_floor() before consuming each AE
* — mirroring ha_sync_raft_log_from_recovery's own boot-time computation,
* but applied live, on the same single AE-receive call path (no new lock,
* no cross-thread mutation) — so a receiver whose OWN local checkpoint
* write left log_len stale (bug-1466, deliberately NOT fixed at its root
* by this change) still correctly logOk-accepts the checkpoint's own lsn
* as a valid predecessor.
* - TERM-INDEX side (persist.c, record_entry_term/entry_term_locked):
* a THIRD, previously-undiscovered gap found by live instrumentation of
* THIS test after the first two fixes above still left it RED. The first
* two fixes both rely on `persist_entry_term(checkpoint_lsn)` staying
* "absent" (0) FOREVER on every member (documented design intent in both
* comments above and in persist_write_checkpoint_marker's own comment).
* That was false: record_entry_term's geometric-growth realloc zero-filled
* any skipped gap slot (e.g. the checkpoint's own, deliberately-unrecorded
* LSN), and entry_term_locked's `lsn <= term_index_count` presence check
* then read that zero-fill back as WAL_LEGACY_TERM (1), NOT absent (0),
* the INSTANT any LATER lsn was recorded and grew term_index_count past
* the gap — which happens immediately in this test (and in any live
* cluster) once the leader appends its next real entry after the
* checkpoint. This made the checkpoint LSN's "term" a TIME-DEPENDENT
* value: a fresh read right after the checkpoint returned 0 (absent, as
* designed), but the SAME read after a later entry was appended returned
* 1 — so the sender (which streams AFTER its own later entries exist)
* computed prev_log_term=1 for the entry right after the checkpoint,
* while a receiver that had not yet appended anything past its own
* checkpoint still (correctly, per design) read 0 for the same LSN —
* logOk's equality check then spuriously mismatched (0 != 1) and rejected
* forever. Fixed with a TERM_INDEX_UNSET sentinel (UINT64_MAX) that marks
* a never-explicitly-written slot as absent PERMANENTLY, independent of
* later array growth — see TERM_INDEX_UNSET's comment in persist.c for
* the full mechanism. Confirmed via real fprintf instrumentation at the
* streamer, the JIT resync, and raft.c's logOk check (added and removed
* during this investigation) that prev_log_term flipped from 0 to 1
* exactly at the point lsn=4 was appended, before the persist.c fix; the
* same instrumentation confirmed prev_log_term=0 on both sides and
* log_ok=1 after the fix.
*
* Fixture: a REAL leader persist_t WAL [lsn=1 real, lsn=2 WAL_CHECKPOINT
* (via the real persist_checkpoint()), lsn=3 real, lsn=4 real]. A REAL
* receiver cluster_t independently applies the SAME committed prefix through
* lsn=1 and independently writes its OWN local checkpoint at lsn=2 (exactly
* mirroring persist_write_checkpoint_marker's deterministic per-member
* trigger — every member that applies a committed WAL_CONFIG_MARKER
* checkpoints at the identical LSN, not just the leader). The receiver's
* raft.log_len is left at its STALE, pre-checkpoint value (1) — the exact
* live-write-time gap bug-1466 documents (persist_write_checkpoint_marker
* bypasses raft_log_append, so log_len never advances for a checkpoint on
* its own) — this is the precondition the permanent stall needs.
*
* ha_stream_wal_to_standby streams (standby_lsn=2, committed_lsn=4] and a
* capture-and-replay hook feeds every batch into the REAL
* ha_apply_wal_catchup_entry receive entry point against the receiver —
* exactly the wire hop xce_decode → ha_apply_wal_catchup_entry performs live.
*
* RED (pre-fix, verified live during this investigation by reverting the
* persist.c term-index sentinel fix with the other two parts still in
* place): the receiver
* gets permanently stuck AT OR BEFORE the checkpoint boundary — lsn=3's
* prev_log_index is computed one below where the checkpoint actually left
* the receiver's log, so ha_apply_term_entry_via_library's accept-check
* (persist_entry_term(p, e->lsn) against the ACTUAL write position) never
* matches, no ACK is sent, and every subsequent entry's prev_log_index only
* grows further past the receiver's true (stale) log_len — ae_reject_
* logmismatch forever, zero progress. A second stream of the IDENTICAL range
* makes IDENTICAL zero progress, proving the wedge is permanent, not a
* transient one-round lag.
*
* GREEN (post-fix): the receiver catches all the way up to lsn=4.
* ══════════════════════════════════════════════════════════════════════════ */
/* Bring a node up as a raft FOLLOWER at `term` (mirrors t42_make_leader,
* make_follower in test_term_safety_adapter.c — this file has no standby-role
* helper of its own yet). */
static void t_bug1475_make_follower(cluster_t *c, uint64_t term, int n_peers)
{
atomic_store(&c->ha.role, CTRL_STANDBY);
atomic_store(&c->ha.term, term);
ha_init_raft(&c->ha, c, n_peers);
c->ha.raft.term = term;
c->ha.raft.role = RAFT_FOLLOWER;
}
/* Replay-into-receiver hook: feeds every streamed batch into the REAL
* ha_apply_wal_catchup_entry receive path against g_bug1475_recv, exactly as
* the wire hop (xce_decode) would on a live connection. */
static cluster_t *g_bug1475_recv = NULL;
static int bug1475_replay_hook(endpoint_t *ep, xce_request_t *req)
{
(void)ep;
if (!req || !req->body.walcatchupentrymsg || !g_bug1475_recv)
return 0;
wal_catchup_entry_t *batch =
(wal_catchup_entry_t *)req->body.walcatchupentrymsg;
ha_apply_wal_catchup_entry(&g_bug1475_recv->ha, &g_bug1475_recv->persist,
batch, /*ep*/NULL, g_bug1475_recv);
return 0;
}
static void test_bug1475_catchup_survives_midstream_checkpoint(void)
{
char ltmp[256], rtmp[256];
strcpy(ltmp, "/tmp/xce_cr_b1475_leader_XXXXXX");
strcpy(rtmp, "/tmp/xce_cr_b1475_recv_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(ltmp));
TEST_ASSERT_NOT_NULL(mkdtemp(rtmp));
cluster_t *leader = t42_make_cluster(ltmp);
cluster_t *recv = t42_make_cluster(rtmp);
TEST_ASSERT_NOT_NULL(leader);
TEST_ASSERT_NOT_NULL(recv);
enum { TERM = 3 };
t42_make_leader(leader, TERM, /*n_peers=*/3);
t_bug1475_make_follower(recv, TERM, /*n_peers=*/3);
/* Identical prefix on both members through lsn=1. persist_wal_async is a
* raw WAL write and does not itself update raft.log_len (that only
* happens via ha_primary_log_append/raft_log_append on the real commit
* path) -- both members already correctly hold index 1 (e.g. from a
* prior, successful AE exchange this test does not need to re-simulate),
* so both are seeded here to match. */
persist_set_term(&leader->persist, TERM);
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(&leader->persist, WAL_JOB_UPDATE, NULL, 0)); /* lsn=1 */
leader->ha.raft.log_len = 1;
persist_set_term(&recv->persist, TERM);
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(&recv->persist, WAL_JOB_UPDATE, NULL, 0)); /* lsn=1 */
recv->ha.raft.log_len = 1;
/* Each member independently checkpoints at lsn=2 -- the SAME deterministic
* side effect persist_checkpoint_marker_triggered fires on every member
* that applies a committed WAL_CONFIG_MARKER, not just the leader. Force a
* synchronous flush right after so persist_durable_floor() (which the
* ha.c bug-1475 fix reads) reflects it immediately rather than waiting on
* the background fsync_worker's own timer (cfg.wal_sync_ms=5000). */
TEST_ASSERT_EQUAL_INT(0, persist_checkpoint(&leader->persist)); /* lsn=2 */
TEST_ASSERT_EQUAL_INT(0, persist_flush(&leader->persist));
TEST_ASSERT_EQUAL_INT(0, persist_checkpoint(&recv->persist)); /* lsn=2 */
TEST_ASSERT_EQUAL_INT(0, persist_flush(&recv->persist));
/* bug-1466 (unfixed, deliberately NOT touched by this fix): the checkpoint
* write bypassed raft_log_append on BOTH members, so log_len stayed at 1
* on both even though current_lsn moved to 3 -- exactly the precondition
* the live permanent stall needs. */
TEST_ASSERT_EQUAL_UINT64(1, leader->ha.raft.log_len);
TEST_ASSERT_EQUAL_UINT64(1, recv->ha.raft.log_len);
TEST_ASSERT_EQUAL_UINT64(3, (uint64_t)leader->persist.current_lsn);
TEST_ASSERT_EQUAL_UINT64(3, (uint64_t)recv->persist.current_lsn);
/* Leader appends two more REAL entries after the checkpoint. */
persist_set_term(&leader->persist, TERM);
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(&leader->persist, WAL_JOB_UPDATE, NULL, 0)); /* lsn=3 */
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(&leader->persist, WAL_JOB_UPDATE, NULL, 0)); /* lsn=4 */
atomic_store_explicit(&leader->persist.committed_lsn, 4,
memory_order_release);
/* Drive the REAL catch-up streamer -> REAL receive path. standby_lsn=2:
* this receiver already durably holds through the checkpoint's own lsn. */
g_bug1475_recv = recv;
xce_wal_stream_request_hook = bug1475_replay_hook;
int rc = ha_stream_wal_to_standby(/*ep=*/NULL, leader->persist.wal_path,
&leader->persist, /*leader_term=*/TERM,
/*standby_lsn=*/2, /*committed_lsn=*/4);
xce_wal_stream_request_hook = NULL;
g_bug1475_recv = NULL;
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc, "ha_stream_wal_to_standby failed");
uint64_t applied = atomic_load_explicit(&recv->ha.applied_lsn,
memory_order_acquire);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(4, applied,
"bug-1475: the standby must catch all the way up to lsn=4 across the "
"mid-stream checkpoint, not stall permanently at/before it");
TEST_ASSERT_EQUAL_UINT64(4, recv->ha.raft.log_len);
TEST_ASSERT_EQUAL_UINT64(TERM, persist_entry_term(&recv->persist, 4));
/* A SECOND identical stream (the real heartbeat-driven retry) must be a
* clean no-op / re-confirmation, never regress applied_lsn -- proving
* this is genuine convergence, not a one-round fluke. */
g_bug1475_recv = recv;
xce_wal_stream_request_hook = bug1475_replay_hook;
rc = ha_stream_wal_to_standby(/*ep=*/NULL, leader->persist.wal_path,
&leader->persist, /*leader_term=*/TERM,
/*standby_lsn=*/4, /*committed_lsn=*/4);
xce_wal_stream_request_hook = NULL;
g_bug1475_recv = NULL;
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_UINT64(4, atomic_load_explicit(&recv->ha.applied_lsn,
memory_order_acquire));
t42_teardown(leader);
t42_teardown(recv);
printf("test_bug1475_catchup_survives_midstream_checkpoint: PASS\n");
}
/*
* bug-1028 — the catch-up streamer must source each streamed entry's term and
* the AppendEntries prev_log_term from the AUTHORITATIVE per-LSN term index
@ -4687,6 +4917,219 @@ static void test_snapshot_transfer_two_chunks(void)
printf("test_snapshot_transfer_two_chunks: PASS\n");
}
/* ══════════════════════════════════════════════════════════════════════════
* bug-1475 Finding 2 (coordinator review, 2026-07-27/28): the SAME permanent
* catch-up wedge bug-1475 fixed for WAL-streaming catch-up, reachable
* instead via the SNAPSHOT-INSTALL path (HA_REPL_SEND_SNAPSHOT — dispatched
* whenever a standby has fallen too far behind for incremental streaming, a
* live, common path, not dead code).
*
* ha_snapshot_do_send (ha.c) computes checkpoint_term = persist_entry_term(
* leader, checkpoint_lsn) — which, thanks to bug-1475's TERM_INDEX_UNSET
* fix, is now PERMANENTLY 0 (absent) for any real checkpoint LSN
* (persist_write_checkpoint_marker never records a term for its own entry,
* by design). Before THIS fix, ha_handle_snapshot_chunk's
* persist_reconcile_checkpoint_term(p, checkpoint_lsn, checkpoint_term)
* wrote that 0 RAW into the term index via an EXPLICIT record_entry_term
* call (not a growth-fill) — so entry_term_locked's legacy-0 rule read it
* back as WAL_LEGACY_TERM(1), NOT absent, permanently diverging from the
* leader's own permanently-absent(0) value for the IDENTICAL LSN. A
* subsequent AppendEntries whose prev_log_index==checkpoint_lsn (the entry
* immediately following it) then mismatched logOk (0 != 1) forever.
*
* RED (pre-fix, verified by temporarily reverting persist_reconcile_
* checkpoint_term's TERM_INDEX_UNSET substitution): the receiver's
* checkpoint-boundary term reads back 1 (WAL_LEGACY_TERM), not 0, and the
* subsequent AppendEntries is rejected — applied_lsn never reaches 3.
* GREEN (post-fix): the receiver's checkpoint-boundary term reads back 0
* (absent, matching the leader), and the subsequent AppendEntries is
* accepted.
* ══════════════════════════════════════════════════════════════════════════ */
static cluster_t *g_bug1475f2_recv = NULL;
static int bug1475f2_snapshot_replay_hook(endpoint_t *ep, xce_request_t *req)
{
(void)ep;
if (!req || !req->body.snapshottransfermsg || !g_bug1475f2_recv)
return 0;
snapshot_transfer_t *chunk =
(snapshot_transfer_t *)req->body.snapshottransfermsg;
return ha_handle_snapshot_chunk(&g_bug1475f2_recv->ha,
&g_bug1475f2_recv->persist,
g_bug1475f2_recv->persist.snap_path,
chunk, chunk->raw_payload,
chunk->raw_payload_len);
}
static void test_bug1475_finding2_snapshot_install_checkpoint_term_absent(void)
{
char ltmp[256], rtmp[256];
strcpy(ltmp, "/tmp/xce_cr_b1475f2_leader_XXXXXX");
strcpy(rtmp, "/tmp/xce_cr_b1475f2_recv_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(ltmp));
TEST_ASSERT_NOT_NULL(mkdtemp(rtmp));
cluster_t *leader = t42_make_cluster(ltmp);
cluster_t *recv = t42_make_cluster(rtmp);
TEST_ASSERT_NOT_NULL(leader);
TEST_ASSERT_NOT_NULL(recv);
extern int (*xce_snapshot_chunk_hook)(endpoint_t *, xce_request_t *);
extern int ha_send_snapshot_to_standby(endpoint_t *, const char *,
uint64_t, uint64_t);
enum { TERM = 5 };
t42_make_leader(leader, TERM, /*n_peers=*/3);
t_bug1475_make_follower(recv, TERM, /*n_peers=*/3);
/* ha_handle_snapshot_chunk takes ha->lock directly; t42_make_cluster /
* ha_init_raft never call pthread_mutex_init on it (only the AE-receive
* path is exercised by the OTHER bug-1475 test, which never locks it) —
* initialize it explicitly here rather than relying on a zeroed mutex
* happening to behave like PTHREAD_MUTEX_INITIALIZER. */
pthread_mutex_init(&recv->ha.lock, NULL);
/* Leader: a real entry at lsn=1, then a REAL checkpoint at lsn=2 via
* persist_checkpoint -> persist_write_checkpoint_marker (the exact
* production path — not a synthetic marker).
*
* p->checkpoint_lsn is NOT the WAL_CHECKPOINT entry's own physical lsn —
* persist_checkpoint() (the general/periodic form ha_snapshot_do_send
* actually calls) labels the snapshot with last_applied_lsn, read
* immediately AFTER persist_write_checkpoint_marker has already
* consumed the next lsn slot for the checkpoint itself. In a
* continuously-active system checkpoint_lsn is usually one BELOW the
* checkpoint's own physical lsn (the last real entry applied so far);
* Finding 2 requires the specific (real, reachable) quiescent-state
* timing where the apply pump has already walked all the way through
* and landed exactly ON the checkpoint's own lsn (persist_apply_pump_run
* advances last_applied_lsn one lsn at a time, INCLUDING a no-op pass
* over a WAL_CHECKPOINT slot, before this snapshot-send cycle runs) —
* e.g. right after a config upgrade with no immediate follow-up
* activity. Seed that precondition directly (deterministic; the real
* race depends on independent apply-pump timing this test does not
* need to reproduce) so persist_checkpoint()'s OWN checkpoint_lsn
* coincides with the checkpoint entry's own un-recorded lsn — exactly
* the case where persist_entry_term(checkpoint_lsn) reads genuinely
* absent. */
persist_set_term(&leader->persist, TERM);
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(&leader->persist, WAL_JOB_UPDATE, NULL, 0)); /* lsn=1 */
TEST_ASSERT_EQUAL_UINT64(2, (uint64_t)leader->persist.current_lsn);
atomic_store_explicit(&leader->persist.last_applied_lsn, 2,
memory_order_release);
TEST_ASSERT_EQUAL_INT(0, persist_checkpoint(&leader->persist)); /* lsn=2 */
TEST_ASSERT_EQUAL_INT(0, persist_flush(&leader->persist));
TEST_ASSERT_EQUAL_UINT64_MESSAGE(2, leader->persist.checkpoint_lsn,
"precondition: checkpoint_lsn must coincide with the checkpoint "
"entry's own physical lsn for this test to exercise Finding 2 "
"(otherwise it would read a real, non-absent term instead)");
/* Leader appends one more real entry AFTER the checkpoint — mirrors the
* real timing that exposed Finding 2 (persist_entry_term(checkpoint_lsn)
* read right after the checkpoint vs. after later growth exists). */
persist_set_term(&leader->persist, TERM);
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(&leader->persist, WAL_JOB_UPDATE, NULL, 0)); /* lsn=3 */
/* Precondition: exactly what ha_snapshot_do_send's real computation
* would read for the wire — genuinely absent (0), permanently, even
* after later growth (TERM_INDEX_UNSET's whole point). */
uint64_t checkpoint_term =
persist_entry_term(&leader->persist, leader->persist.checkpoint_lsn);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, checkpoint_term,
"precondition: a real checkpoint LSN's term reads absent (0) on the "
"sender, even after later entries exist — the exact value "
"ha_snapshot_do_send would put on the wire");
/* Drive the REAL snapshot send -> receive path: ha_send_snapshot_to_standby
* chunks the on-disk snapshot file (written by persist_checkpoint above)
* and stamps checkpoint_term on every chunk; the capture-and-replay hook
* feeds each chunk into the REAL ha_handle_snapshot_chunk on the
* receiver, exactly the wire hop a live transfer performs. */
g_bug1475f2_recv = recv;
xce_snapshot_chunk_hook = bug1475f2_snapshot_replay_hook;
int rc = ha_send_snapshot_to_standby(/*ep=*/NULL, leader->persist.snap_path,
leader->persist.checkpoint_lsn,
checkpoint_term);
xce_snapshot_chunk_hook = NULL;
g_bug1475f2_recv = NULL;
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc, "ha_send_snapshot_to_standby failed");
TEST_ASSERT_EQUAL_UINT64(leader->persist.checkpoint_lsn,
recv->persist.checkpoint_lsn);
/* The receiver must reconcile the checkpoint boundary LSN's term to
* match the leader's — genuinely absent (0), NOT WAL_LEGACY_TERM(1). */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0,
persist_entry_term(&recv->persist, leader->persist.checkpoint_lsn),
"bug-1475 Finding 2: the receiver's checkpoint-boundary term must "
"reconcile to absent (0), matching the leader's permanently-absent "
"value for the SAME lsn — NOT get reinterpreted as "
"WAL_LEGACY_TERM(1) by persist_reconcile_checkpoint_term's "
"record_entry_term(..., 0) call");
/* Isolate Finding 2's term-index concern from two OTHER, SEPARATE,
* pre-existing (unchanged by this fix) mechanics this test is not about:
*
* 1. ha_handle_snapshot_chunk does not itself resync ha->raft.log_len
* (only ha_apply_term_entry_via_library's JIT resync does, on the
* NEXT AE) — set it directly to the checkpoint's own lsn here, the
* value a real deployment reaches once durable_lsn catches up (or a
* later boot-reseed runs).
* 2. load_snapshot() sets p->current_lsn = stored_lsn (the checkpoint's
* OWN lsn), NOT stored_lsn + 1 — confirmed via direct instrumentation
* (current_lsn read back as 2, matching checkpoint_lsn, immediately
* after this install). persist_wal_write_replicated's forward-gap
* guard (lsn > current_lsn -> EAGAIN) would then reject ANY entry at
* checkpoint_lsn+1 regardless of term, confounding this test's
* specific term question. Whether that current_lsn convention
* correctly bridges into the next live catch-up cycle in production
* is a DIFFERENT, pre-existing question this test does not answer —
* flagged here, not fixed (out of Finding 2's scope: the coordinator's
* review identified a term(0-vs-1) mismatch specifically, not a
* current_lsn contiguity gap). Bypass it directly so only the term
* question is under test. */
recv->ha.raft.log_len = leader->persist.checkpoint_lsn;
recv->persist.current_lsn = leader->persist.checkpoint_lsn + 1;
/* Drive a REAL subsequent AppendEntries whose prev_log_index is the
* checkpoint's own LSN — the entry immediately following it. */
wal_catchup_entry_t sub;
memset(&sub, 0, sizeof(sub));
sub.lsn = 3;
sub.entry_type = WAL_JOB_UPDATE;
sub.term = TERM;
sub.prev_log_index = leader->persist.checkpoint_lsn; /* == 2 */
sub.prev_log_term = checkpoint_term; /* == 0, absent */
sub.leader_current_term = TERM;
wal_catchup_entry_t *sub_arr[1] = { &sub };
wal_catchup_entry_t batch;
memset(&batch, 0, sizeof(batch));
batch.n_entries = 1;
batch.entries = sub_arr;
batch.term = TERM;
batch.prev_log_index = sub.prev_log_index;
batch.prev_log_term = sub.prev_log_term;
rc = ha_apply_wal_catchup_entry(&recv->ha, &recv->persist, &batch,
/*ep*/NULL, recv);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(3,
atomic_load_explicit(&recv->ha.applied_lsn, memory_order_acquire),
"bug-1475 Finding 2: the standby must accept the AppendEntries "
"immediately following the checkpoint boundary after a snapshot "
"install, not permanently reject it (ae_reject_logmismatch) due to "
"a term(0) vs term(1) mismatch at the checkpoint LSN");
TEST_ASSERT_EQUAL_UINT64(TERM, persist_entry_term(&recv->persist, 3));
pthread_mutex_destroy(&recv->ha.lock);
t42_teardown(leader);
t42_teardown(recv);
printf("test_bug1475_finding2_snapshot_install_checkpoint_term_absent: PASS\n");
}
static void test_snapshot_transfer_small_less_than_one_chunk(void)
{
char tmpdir[] = "/tmp/xcectest_snaptrans_XXXXXX";
@ -5964,9 +6407,11 @@ void test_config_repl(void)
RUN_TEST(test_wal_stream_partial_batch);
RUN_TEST(test_wal_stream_empty_range_no_batches);
RUN_TEST(test_wal_stream_skips_checkpoint_marker);
RUN_TEST(test_bug1475_catchup_survives_midstream_checkpoint); /* bug-1475 */
RUN_TEST(test_wal_stream_term_from_index_not_raw_hdr); /* bug-1028 */
/* Task 3.5: snapshot chunk transfer (sender side) */
RUN_TEST(test_snapshot_transfer_two_chunks);
RUN_TEST(test_bug1475_finding2_snapshot_install_checkpoint_term_absent); /* bug-1475 Finding 2 */
RUN_TEST(test_snapshot_transfer_small_less_than_one_chunk);
RUN_TEST(test_snapshot_transfer_stops_and_propagates_send_failure);
/* Task 3.5c: standby snapshot receive handler */