lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha-wal-replication): apply_wal_entry must not own the WAL CRC chain (bug-778)

Browse source 
On the standby replicated-apply path, ha_apply_single_entry wrote the entry via
persist_wal_write_replicated (which correctly set the self-computed CRC chain
link p->last_crc), then called apply_wal_entry with a SYNTHETIC header whose
crc=0. apply_wal_entry's `p->last_crc = hdr->crc` clobbered the chain link with
0, so every subsequent replicated write stamped prev_crc=0 on disk. The on-disk
CRC chain broke at the second replicated entry — invisible while live (standbys
serve in-memory state) but fatal on COLD RESTART: replay hit the break, returned
RECOVER_TRUNCATED, recovered only the config prefix, and lost all committed jobs
(joblist=0). The sticky truncated verdict then drove a promotion-fence
availability wedge (bug-777) when every survivor was truncated.

Fix — single ownership of the WAL header + CRC chain:
- apply_wal_entry is now LOGICAL entity-restore only: signature
  (p, type, lsn, payload, payload_len); no header. It advances the current_lsn
  high-water (monotonic) but NEVER touches p->last_crc.
- The physical layer is the sole owner of the CRC chain: persist_wal_write_-
  replicated on the live path, replay_wal/persist_recover on recovery.
- ha_apply_single_entry drops the fabricated header and passes entry fields
  directly. The persist_recover replay caller is updated to the new signature.

Verified: new RED->GREEN test test_replicated_apply_chain_survives_cold_restart
(drives the real ha_apply_wal_catchup_entry path, cold-replays -> RECOVER_CLEAN,
all N recovered); controller suite 0 failures; 3-node rolling-SIGKILL soak on a
real ext2/3 fs now passes 5/5 (was wedge+loss at round 3); full cold restart
recovers all 6 committed jobs (was 0) with CLEAN verdicts and a normal election.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-06-19 14:12:53 +00:00
parent 276e26c37b
commit 64ab210cbf
7 changed files with 136 additions and 45 deletions
Download patch file Download diff file Expand all files Collapse all files

14
src/controller/ha.c
View file

@ -1578,14 +1578,12 @@ static int ha_apply_single_entry(persist_t *p, wal_catchup_entry_t *e)
return prc;
}
wal_entry_hdr_t hdr;
memset(&hdr, 0, sizeof(hdr));
hdr.lsn = e->lsn;
hdr.type = (uint32_t)e->entry_type;
hdr.payload_len = (uint32_t)e->raw_payload_len;
hdr.crc = 0;
hdr.prev_crc = 0;
return apply_wal_entry(p, &hdr, e->raw_payload);
/* Logical entity restore only. persist_wal_write_replicated (above) already
* wrote the entry and owns the on-disk CRC chain + current_lsn — so we pass
* the entry fields directly rather than fabricating a header. (bug-778:
* a synthetic crc=0 header here used to clobber p->last_crc.) */
return apply_wal_entry(p, (uint32_t)e->entry_type, e->lsn,
e->raw_payload, (uint32_t)e->raw_payload_len);
}
/* ha_send_wal_ack_reply: send WalAckReply {term, lsn, success=1} back to