lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Browse source 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions
Download patch file Download diff file Expand all files Collapse all files

65
src/lib/testsuite/test_wal_wire.c
View file

@ -90,6 +90,70 @@ static void test_wal_batch_roundtrip_payload_bytes(void)
printf("test_wal_batch_roundtrip_payload_bytes: PASS\n");
}
/* Production-scale batch: many entries each carrying a full-size (872-byte)
* capnp-Job-sized payload — mirrors the live catch-up stream where ctrl-1
* rejoined and received header-only entries (PLEN=0). The small 2-entry
* round-trip passes; this reproduces whatever the larger/multi-segment batch
* does on the wire (bug-847). */
static void test_wal_batch_roundtrip_large_payloads(void)
{
/* KNOWN-FAILING REPRODUCER for bug-847: a production-scale batch that spans
* more than one capnp segment loses its per-entry payload Data pointers on
* the wire (decode returns raw_payload_len=0; it also leaks the orphaned
* decode allocations under ASAN). IGNORE'd so the suite stays green; REMOVE
* this TEST_IGNORE line when bug-847 is fixed and assert the payloads
* survive. Decisive evidence: this fails "Expected 872 Was 0" while the
* 2-entry round-trip above passes. */
TEST_IGNORE_MESSAGE(
"bug-847: catch-up batch drops payloads across capnp segments — "
"un-ignore when fixed");
enum { N = 14, PSZ = 872 };
wal_catchup_entry_t e[N];
wal_catchup_entry_t *entries[N];
uint8_t* payloads[N];
for (int i = 0; i < N; i++) {
payloads[i] = malloc(PSZ);
TEST_ASSERT_NOT_NULL(payloads[i]);
/* distinct, non-zero content per entry so a dropped/zeroed payload is
* unambiguous */
for (int b = 0; b < PSZ; b++)
payloads[i][b] = (uint8_t)((i * 31 + b) & 0xff);
memset(&e[i], 0, sizeof(e[i]));
e[i].lsn = (uint64_t)(i + 1);
e[i].entry_type = 1; /* WAL_JOB_SUBMIT */
e[i].raw_payload = payloads[i];
e[i].raw_payload_len = PSZ;
entries[i] = &e[i];
}
io_packet_t* pkt = encode_batch(entries, N);
xce_request_t req;
memset(&req, 0, sizeof(req));
TEST_ASSERT_EQUAL_INT(0, xce_decode(pkt, &req));
wal_catchup_entry_t* batch = req.body.walcatchupentrymsg;
TEST_ASSERT_NOT_NULL(batch);
TEST_ASSERT_EQUAL_INT(N, batch->n_entries);
for (int i = 0; i < N; i++) {
TEST_ASSERT_NOT_NULL(batch->entries[i]);
TEST_ASSERT_EQUAL_UINT64((uint64_t)(i + 1), batch->entries[i]->lsn);
/* the bug: these come back NULL / len 0 on the live path */
TEST_ASSERT_EQUAL_size_t(PSZ, batch->entries[i]->raw_payload_len);
TEST_ASSERT_NOT_NULL(batch->entries[i]->raw_payload);
TEST_ASSERT_EQUAL_MEMORY(payloads[i], batch->entries[i]->raw_payload,
PSZ);
}
xce_free_request(&req);
for (int i = 0; i < N; i++)
free(payloads[i]);
printf("test_wal_batch_roundtrip_large_payloads: PASS\n");
}
static void test_wal_batch_entry_without_payload(void)
{
/* An entry with no payload (e.g. header-only types) must decode with
@ -215,6 +279,7 @@ static void test_snapshot_chunk_empty_payload(void)
void test_wal_wire(void)
{
RUN_TEST(test_wal_batch_roundtrip_payload_bytes);
RUN_TEST(test_wal_batch_roundtrip_large_payloads);
RUN_TEST(test_wal_batch_entry_without_payload);
RUN_TEST(test_wal_batch_encode_rejects_invalid_args);
RUN_TEST(test_snapshot_chunk_roundtrip);