wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(ha-wal-replication): apply_wal_entry must not own the WAL CRC chain (bug-778)

Browse source 
On the standby replicated-apply path, ha_apply_single_entry wrote the entry via
persist_wal_write_replicated (which correctly set the self-computed CRC chain
link p->last_crc), then called apply_wal_entry with a SYNTHETIC header whose
crc=0. apply_wal_entry's `p->last_crc = hdr->crc` clobbered the chain link with
0, so every subsequent replicated write stamped prev_crc=0 on disk. The on-disk
CRC chain broke at the second replicated entry — invisible while live (standbys
serve in-memory state) but fatal on COLD RESTART: replay hit the break, returned
RECOVER_TRUNCATED, recovered only the config prefix, and lost all committed jobs
(joblist=0). The sticky truncated verdict then drove a promotion-fence
availability wedge (bug-777) when every survivor was truncated.

Fix — single ownership of the WAL header + CRC chain:
- apply_wal_entry is now LOGICAL entity-restore only: signature
  (p, type, lsn, payload, payload_len); no header. It advances the current_lsn
  high-water (monotonic) but NEVER touches p->last_crc.
- The physical layer is the sole owner of the CRC chain: persist_wal_write_-
  replicated on the live path, replay_wal/persist_recover on recovery.
- ha_apply_single_entry drops the fabricated header and passes entry fields
  directly. The persist_recover replay caller is updated to the new signature.

Verified: new RED->GREEN test test_replicated_apply_chain_survives_cold_restart
(drives the real ha_apply_wal_catchup_entry path, cold-replays -> RECOVER_CLEAN,
all N recovered); controller suite 0 failures; 3-node rolling-SIGKILL soak on a
real ext2/3 fs now passes 5/5 (was wedge+loss at round 3); full cold restart
recovers all 6 committed jobs (was 0) with CLEAN verdicts and a normal election.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-06-19 14:12:53 +00:00
parent 276e26c37b
commit 64ab210cbf
7 changed files with 136 additions and 45 deletions
Download patch file Download diff file Expand all files Collapse all files

20
src/controller/testsuite/test_repl_apply.c
View file

@ -190,14 +190,16 @@ static void test_apply_wal_entry_advances_lsn(void)
uint8_t payload[4] = {1, 2, 3, 4};
int rc = apply_wal_entry(p, &hdr, payload);
int rc = apply_wal_entry(p, hdr.type, hdr.lsn, payload, hdr.payload_len);
TEST_ASSERT_EQUAL_INT(0, rc);
/* current_lsn must have advanced to at least lsn+1 */
/* current_lsn high-water must have advanced to at least lsn+1 */
TEST_ASSERT_TRUE(p->current_lsn > 42);
/* last_crc must reflect the entry's crc */
TEST_ASSERT_EQUAL_UINT32(0xDEADBEEF, p->last_crc);
/* bug-778: apply_wal_entry is LOGICAL restore only — it must NOT touch the
* on-disk CRC chain link. last_crc stays at its initial value (the physical
* writer / replay owns it); it must NOT adopt the header's crc. */
TEST_ASSERT_EQUAL_UINT32(WAL_CRC_NONE, p->last_crc);
destroy_test_persist(p);
printf("test_apply_wal_entry_advances_lsn: PASS\n");
@ -215,7 +217,7 @@ static void test_apply_wal_entry_skips_checkpoint(void)
hdr.term = 1;
hdr.payload_len = 0;
int rc = apply_wal_entry(p, &hdr, NULL);
int rc = apply_wal_entry(p, hdr.type, hdr.lsn, NULL, hdr.payload_len);
TEST_ASSERT_EQUAL_INT(0, rc); /* skip = success, not error */
/* Checkpoint entries are local-only markers — standbys never receive them.
@ -237,7 +239,7 @@ static void test_apply_wal_entry_rejects_oversized(void)
hdr.term = 1;
hdr.payload_len = MAX_WAL_PAYLOAD_BYTES + 1; /* too large */
int rc = apply_wal_entry(p, &hdr, NULL);
int rc = apply_wal_entry(p, hdr.type, hdr.lsn, NULL, hdr.payload_len);
TEST_ASSERT_NOT_EQUAL(0, rc); /* must be rejected */
destroy_test_persist(p);
@ -260,7 +262,7 @@ static void test_apply_wal_entry_skips_unknown_type(void)
hdr.term = 1;
hdr.payload_len = 0;
int rc = apply_wal_entry(p, &hdr, NULL);
int rc = apply_wal_entry(p, hdr.type, hdr.lsn, NULL, hdr.payload_len);
TEST_ASSERT_EQUAL_INT(0, rc); /* unknown type: skip not error */
/* LSN must NOT advance for a skipped-unknown type */
@ -272,8 +274,8 @@ static void test_apply_wal_entry_skips_unknown_type(void)
static void test_apply_wal_entry_null_guard(void)
{
/* Both NULL inputs must return non-zero (EINVAL) */
int rc = apply_wal_entry(NULL, NULL, NULL);
/* NULL persist must return non-zero (EINVAL) */
int rc = apply_wal_entry(NULL, WAL_JOB_UPDATE, 1, NULL, 0);
TEST_ASSERT_NOT_EQUAL(0, rc);
printf("test_apply_wal_entry_null_guard: PASS\n");
}