lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha-term-safety): remove task-2.4 commit min-ceiling (Option A) + fix bug-991 catch-up off-by-one

Browse source 
Two coupled fixes for a live commit-stall (committed_lsn pinned at the own-term
NOOP while a 3-node cluster was durable to LSN 5 — jobs acked but never committed):

bug-991: the WAL catch-up applied_lsn was advanced to lsn+1 (next-to-apply) in
the three live/catch-up apply paths, but recovery/checkpoint/init and the leader's
catch-up streamer treat it as last-held. The over-advertise made the streamer skip
an LSN -> library logOk wedge -> no WalAck. Fixed to last-held (= lsn).

Option A (design reconsideration of REVIEW ISSUE 3): removed the
committed = min(durable-majority, raft_commit_index) ceiling. Under single-writer
the durable-majority watermark is already (term,lsn)-current-term-safe (term-carrying
replication + (term,lsn) reconcile/truncate-before-durable + own-term NOOP gate +
post-election truncate-to-safe), so the library commit_index was a redundant,
starvation-prone gate that pinned committed below the safe durable majority whenever
the library lagged. The durable-majority barrier + own-term writable gate are now
authoritative; raft_commit_index is inert-advisory.

Safety: reviewer-verified the durable-majority watermark is (term,lsn)-safe and
Figure-8 holds via the own-term gate (new RED->GREEN test); bug-851 cannot recur.
Live: commit trace 4->5 + walcompare committed-prefix identical across 3 nodes.
177 unit tests pass.
This commit is contained in:
Rongsong Shen 2026-06-23 00:21:06 +00:00
parent df4bc4f86a
commit 83fbed2370
5 changed files with 396 additions and 172 deletions
Download patch file Download diff file Expand all files Collapse all files

53
src/controller/testsuite/test_config_repl.c
View file

@ -2020,8 +2020,11 @@ static void test_wal_catchup_entry_advances_applied_lsn(void)
/* GREEN will implement ha_apply_wal_catchup_entry() */
int rc = ha_apply_wal_catchup_entry(&ha, p, &entry, NULL, NULL);
TEST_ASSERT_EQUAL_INT(0, rc);
/* applied_lsn must advance to lsn+1 = 43 (mirrors persist->current_lsn) */
TEST_ASSERT_EQUAL_UINT64(43,
/* bug-991: applied_lsn is the LAST CONTIGUOUSLY-HELD LSN, so a single applied
* entry at LSN 42 advances applied_lsn to 42 (NOT 43). The +1 over-advertised
* the standby's watermark and made the leader's catch-up streamer skip the
* real next-needed LSN. */
TEST_ASSERT_EQUAL_UINT64(42,
(uint64_t)atomic_load_explicit(&ha.applied_lsn, memory_order_acquire));
ha_fini(&ha);
@ -2128,8 +2131,9 @@ static void test_wal_catchup_batch_advances_applied_lsn(void)
int rc = ha_apply_wal_catchup_entry(&ha, p, &outer, NULL, NULL);
TEST_ASSERT_EQUAL_INT(0, rc);
/* applied_lsn must advance to last_entry->lsn + 1 = 13 */
TEST_ASSERT_EQUAL_UINT64(13,
/* bug-991: applied_lsn = last contiguously-held LSN = last_entry->lsn = 12
* (NOT 13). */
TEST_ASSERT_EQUAL_UINT64(12,
(uint64_t)atomic_load_explicit(&ha.applied_lsn, memory_order_acquire));
ha_fini(&ha);
@ -2175,8 +2179,9 @@ static void test_wal_catchup_partial_batch(void)
int rc = ha_apply_wal_catchup_entry(&ha, p, &outer, NULL, NULL);
TEST_ASSERT_EQUAL_INT(0, rc);
/* applied_lsn must advance to 21+1=22 */
TEST_ASSERT_EQUAL_UINT64(22,
/* bug-991: applied_lsn = last contiguously-held LSN = 21 (NOT 22). Monotonic:
* advances from the pre-set 5 to 21. */
TEST_ASSERT_EQUAL_UINT64(21,
(uint64_t)atomic_load_explicit(&ha.applied_lsn, memory_order_acquire));
ha_fini(&ha);
@ -2225,10 +2230,12 @@ static void test_wal_catchup_batch_error_entry_skipped(void)
/* Overall rc == 0: outer batch call succeeds even with inner skips */
TEST_ASSERT_EQUAL_INT(0, rc);
/* applied_lsn must reach at least 31 (e0 applied); ideally 33 (e2 applied) */
/* bug-991: applied_lsn = last contiguously-held LSN. e0 (LSN 30) applies,
* the NULL sub-entry is skipped, e2 (LSN 32) applies → last-held is 32 (or at
* least 30 if e2 were dropped). Must reach at least 30 (e0 held). */
uint64_t final_lsn = (uint64_t)atomic_load_explicit(&ha.applied_lsn,
memory_order_acquire);
TEST_ASSERT_TRUE(final_lsn >= 31);
TEST_ASSERT_TRUE(final_lsn >= 30);
ha_fini(&ha);
destroy_test_persist_catchup(p);
@ -2263,8 +2270,9 @@ static void test_wal_catchup_batch_zero_entries_fallback(void)
int rc = ha_apply_wal_catchup_entry(&ha, p, &entry, NULL, NULL);
TEST_ASSERT_EQUAL_INT(0, rc);
/* Single-entry path: applied_lsn = 50+1 = 51 */
TEST_ASSERT_EQUAL_UINT64(51,
/* bug-991: single-entry path advances applied_lsn to the last-held LSN = 50
* (NOT 51). */
TEST_ASSERT_EQUAL_UINT64(50,
(uint64_t)atomic_load_explicit(&ha.applied_lsn, memory_order_acquire));
ha_fini(&ha);
@ -2321,8 +2329,9 @@ static void test_wal_catchup_batch_256_entries(void)
int rc = ha_apply_wal_catchup_entry(&ha, p, &outer, NULL, NULL);
TEST_ASSERT_EQUAL_INT(0, rc);
/* applied_lsn must advance to last_entry->lsn + 1 = 1000+255+1 = 1256 */
TEST_ASSERT_EQUAL_UINT64(1256,
/* bug-991: applied_lsn = last contiguously-held LSN = last_entry->lsn =
* 1000+255 = 1255 (NOT 1256). */
TEST_ASSERT_EQUAL_UINT64(1255,
(uint64_t)atomic_load_explicit(&ha.applied_lsn, memory_order_acquire));
ha_fini(&ha);
@ -2340,9 +2349,10 @@ static void test_wal_catchup_batch_256_entries(void)
*
* Strategy: start with a pre-set applied_lsn, apply a batch of 4 entries,
* verify that applied_lsn does NOT advance to an intermediate value and IS
* exactly last_entry+1 at the end. We confirm the "only-after-full-batch"
* invariant by checking the exact final value equals last_lsn+1 (not some
* intermediate lsn+1 from a mid-batch advance).
* exactly the last contiguously-held LSN at the end (bug-991: last_lsn, not
* last_lsn+1). We confirm the "only-after-full-batch" invariant by checking the
* exact final value equals last_lsn (not some intermediate held LSN from a
* mid-batch advance).
* Requirements 5.3, 5.5. */
static void test_wal_catchup_batch_applied_lsn_only_after_full_batch(void)
{
@ -2382,17 +2392,16 @@ static void test_wal_catchup_batch_applied_lsn_only_after_full_batch(void)
int rc = ha_apply_wal_catchup_entry(&ha, p, &outer, NULL, NULL);
TEST_ASSERT_EQUAL_INT(0, rc);
/* applied_lsn must be exactly 204 (203+1) — the post-batch value.
* If the implementation advanced after each sub-entry, it would still
* reach 204, but the test for correctness is that the final value
* equals the LAST entry lsn+1, not any intermediate lsn+1.
* Specifically, it must NOT remain at 201 or 202 (partial advance). */
/* bug-991: applied_lsn must be exactly 203 (the last contiguously-held LSN) —
* the post-batch value. The test for correctness is that the final value
* equals the LAST entry's lsn, not any intermediate held LSN.
* Specifically, it must NOT remain at 200 or 201 (partial advance). */
uint64_t final_lsn = (uint64_t)atomic_load_explicit(&ha.applied_lsn,
memory_order_acquire);
/* Must have advanced past starting value (100) */
TEST_ASSERT_GREATER_THAN_UINT64(100, final_lsn);
/* Must equal the last-entry+1 (204), not any intermediate value */
TEST_ASSERT_EQUAL_UINT64(204, final_lsn);
/* Must equal the last-held LSN (203), not any intermediate value */
TEST_ASSERT_EQUAL_UINT64(203, final_lsn);
ha_fini(&ha);
destroy_test_persist_catchup(p);