fix(ha-wal-replication): apply_wal_entry must not own the WAL CRC chain (bug-778)

On the standby replicated-apply path, ha_apply_single_entry wrote the entry via
persist_wal_write_replicated (which correctly set the self-computed CRC chain
link p->last_crc), then called apply_wal_entry with a SYNTHETIC header whose
crc=0. apply_wal_entry's `p->last_crc = hdr->crc` clobbered the chain link with
0, so every subsequent replicated write stamped prev_crc=0 on disk. The on-disk
CRC chain broke at the second replicated entry — invisible while live (standbys
serve in-memory state) but fatal on COLD RESTART: replay hit the break, returned
RECOVER_TRUNCATED, recovered only the config prefix, and lost all committed jobs
(joblist=0). The sticky truncated verdict then drove a promotion-fence
availability wedge (bug-777) when every survivor was truncated.

Fix — single ownership of the WAL header + CRC chain:
- apply_wal_entry is now LOGICAL entity-restore only: signature
  (p, type, lsn, payload, payload_len); no header. It advances the current_lsn
  high-water (monotonic) but NEVER touches p->last_crc.
- The physical layer is the sole owner of the CRC chain: persist_wal_write_-
  replicated on the live path, replay_wal/persist_recover on recovery.
- ha_apply_single_entry drops the fabricated header and passes entry fields
  directly. The persist_recover replay caller is updated to the new signature.

Verified: new RED->GREEN test test_replicated_apply_chain_survives_cold_restart
(drives the real ha_apply_wal_catchup_entry path, cold-replays -> RECOVER_CLEAN,
all N recovered); controller suite 0 failures; 3-node rolling-SIGKILL soak on a
real ext2/3 fs now passes 5/5 (was wedge+loss at round 3); full cold restart
recovers all 6 committed jobs (was 0) with CLEAN verdicts and a normal election.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-06-19 14:12:53 +00:00
parent 276e26c37b
commit 64ab210cbf
7 changed files with 136 additions and 45 deletions

View file

@ -279,7 +279,7 @@ static void test_noop_apply_advances_lsn_no_entity_effect(void)
hdr.prev_crc = 0;
/* Act: apply the no-op with a NULL payload (header-only) */
int rc = apply_wal_entry(p, &hdr, NULL);
int rc = apply_wal_entry(p, hdr.type, hdr.lsn, NULL, hdr.payload_len);
/* Assert: clean apply, LSN advanced past the entry, nothing else */
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc,
@ -288,7 +288,7 @@ static void test_noop_apply_advances_lsn_no_entity_effect(void)
"applying a WAL_NOOP must advance current_lsn to entry LSN + 1");
/* No cluster → no entity state to inspect; the absence of effect is the
* point. Re-applying must be idempotent on LSN (already advanced). */
int rc2 = apply_wal_entry(p, &hdr, NULL);
int rc2 = apply_wal_entry(p, hdr.type, hdr.lsn, NULL, hdr.payload_len);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc2,
"re-applying a WAL_NOOP must remain a clean no-op");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(noop_lsn + 1, p->current_lsn,