lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Browse source 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions
Download patch file Download diff file Expand all files Collapse all files

62
src/controller/testsuite/test_persist_recovery.c
View file

@ -13,6 +13,7 @@
#include <unity/unity.h>
#include "config.h"
#include "ha.h" /* ha_state_t, ha_sync_applied_lsn_from_recovery (bug-848) */
#include "persist.h"
/* ------------------------------------------------------------------ */
@ -248,6 +249,66 @@ static void test_recover_picks_latest_snapshot(void)
/* Test suite entry point */
/* ------------------------------------------------------------------ */
/* ------------------------------------------------------------------ */
/* bug-848: a recovered member must advertise its REAL last LSN, not 0 */
/* (else the primary re-streams the whole log and overwrites consistent */
/* records instead of fetching only the missing suffix). */
/* ------------------------------------------------------------------ */
static void test_recovered_member_advertises_real_applied_lsn(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_rec848_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
/* Phase 1: write several WAL entries, flush, stop (simulate a node that
* served writes, then was killed). */
{
persist_t* p = make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
for (int i = 0; i < 5; i++)
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_UPDATE, "z", 1));
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
teardown_persist(p);
}
/* Phase 2: restart — fresh persist recovers the WAL; a freshly ha-init'd
* member (applied_lsn == 0) must be synced to the recovered durable tip. */
{
persist_t* p2 = make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p2);
TEST_ASSERT_EQUAL_INT(0, persist_recover(p2));
uint64_t recovered_floor = persist_durable_floor(p2);
TEST_ASSERT_TRUE(recovered_floor > 0); /* WAL was non-empty */
ha_state_t ha;
memset(&ha, 0, sizeof(ha));
atomic_store_explicit(&ha.applied_lsn, 0ULL, memory_order_release);
ha_sync_applied_lsn_from_recovery(&ha, p2);
uint64_t advertised =
atomic_load_explicit(&ha.applied_lsn, memory_order_acquire);
/* The bug: this used to stay 0 → primary re-streams [1..committed]. */
TEST_ASSERT_TRUE(advertised > 0);
TEST_ASSERT_EQUAL_UINT64(recovered_floor, advertised);
/* Monotonic: never lowers an already-higher applied_lsn. */
atomic_store_explicit(&ha.applied_lsn, recovered_floor + 100,
memory_order_release);
ha_sync_applied_lsn_from_recovery(&ha, p2);
TEST_ASSERT_EQUAL_UINT64(
recovered_floor + 100,
atomic_load_explicit(&ha.applied_lsn, memory_order_acquire));
teardown_persist(p2);
}
printf("test_recovered_member_advertises_real_applied_lsn: PASS\n");
}
void test_persist_recovery(void)
{
RUN_TEST(test_recover_from_empty_state);
@ -255,4 +316,5 @@ void test_persist_recovery(void)
RUN_TEST(test_recover_replays_wal_after_checkpoint);
RUN_TEST(test_recover_rejects_corrupt_snapshot);
RUN_TEST(test_recover_picks_latest_snapshot);
RUN_TEST(test_recovered_member_advertises_real_applied_lsn);
}