lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Browse source 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions
Download patch file Download diff file Expand all files Collapse all files

45
src/controller/testsuite/test_repl_apply.c
View file

@ -395,6 +395,50 @@ static void test_repl_write_dedup_idempotent(void)
printf("test_repl_write_dedup_idempotent: PASS\n");
}
/* bug-850 contiguity guard: a replicated entry whose lsn is BEYOND the next-
* expected current_lsn (a forward gap) must be REJECTED with -EAGAIN without
* writing or advancing current_lsn — otherwise it would advance the watermark
* past the hole and the later in-order catch-up of the prefix would be silently
* dropped by the idempotency guard (the failover prefix-loss mechanism). The
* entry becomes writable only once the gap is filled contiguously. */
static void test_repl_write_rejects_forward_gap(void)
{
char tmpdir[256];
persist_t* p = make_test_persist_in(tmpdir, sizeof(tmpdir));
/* Contiguous first entry: lsn 1 == current_lsn → written. */
TEST_ASSERT_EQUAL_INT(0, persist_wal_write_replicated(
p, 1, WAL_JOB_UPDATE, "a", 1));
TEST_ASSERT_EQUAL_UINT64(2, p->current_lsn);
/* Forward gap: lsn 5 > current_lsn(2) — like a live broadcast arriving
* ahead of catch-up. Must be refused (-EAGAIN) and leave current_lsn put. */
int rc_gap = persist_wal_write_replicated(p, 5, WAL_JOB_UPDATE, "e", 1);
TEST_ASSERT_EQUAL_INT(-EAGAIN, rc_gap);
TEST_ASSERT_EQUAL_UINT64(2, p->current_lsn); /* did NOT jump past the gap */
/* Fill the gap in order: 2,3,4 contiguous → accepted. */
for (uint64_t lsn = 2; lsn <= 4; lsn++)
TEST_ASSERT_EQUAL_INT(0, persist_wal_write_replicated(
p, lsn, WAL_JOB_UPDATE, "x", 1));
TEST_ASSERT_EQUAL_UINT64(5, p->current_lsn);
/* Now lsn 5 is contiguous (== current_lsn) → accepted. */
TEST_ASSERT_EQUAL_INT(0, persist_wal_write_replicated(
p, 5, WAL_JOB_UPDATE, "e", 1));
TEST_ASSERT_EQUAL_UINT64(6, p->current_lsn);
destroy_test_persist(p);
/* Recovery yields a clean contiguous chain [1..5] → current_lsn=6. */
persist_t* p2 = reopen_test_persist(tmpdir);
TEST_ASSERT_TRUE(persist_recover(p2) >= 0);
TEST_ASSERT_EQUAL_UINT64(6, p2->current_lsn);
destroy_test_persist(p2);
printf("test_repl_write_rejects_forward_gap: PASS\n");
}
/* WAL_CHECKPOINT must never be persisted via the replicated-write path
* (standbys never receive it; it must be rejected, not written). */
static void test_repl_write_rejects_checkpoint(void)
@ -428,5 +472,6 @@ void test_repl_apply(void)
RUN_TEST(test_repl_write_persists_and_recovers);
RUN_TEST(test_repl_write_crc_chain_recovers);
RUN_TEST(test_repl_write_dedup_idempotent);
RUN_TEST(test_repl_write_rejects_forward_gap);
RUN_TEST(test_repl_write_rejects_checkpoint);
}