lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha-wal-replication): standby durably persists replicated WAL before ACK

Browse source 
A standby applied replicated WAL entries via apply_wal_entry (the replay/
in-memory path) which restores entities + advances applied_lsn but NEVER writes
the entry to the standby's WAL segment. The standby then ACKed (feeding the
primary's commit majority) without persisting — a Raft durability violation:
entries counted toward a commit could be lost on standby restart, and standby
WAL files stayed empty (contradicting steering: log[i] = WAL segments for every
node). This is why ctrl-2/ctrl-3 received + acked the streamed own-term NOOP yet
kept empty seg-*.wal.

Fix: new persist_wal_write_replicated(p, lsn, type, payload, len) writes the
entry to the local WAL at the PRIMARY's lsn (no lsn-reassignment), with a
self-consistent intra-segment CRC chain (the streamed entry carries no term/crc;
each node chains independently and recovery validates intra-segment), fdatasyncs
for durability, advances current_lsn = max(current_lsn, lsn+1), rotates segments,
and does NOT re-broadcast. Wired into ha_apply_single_entry as
persist(durable) -> apply(in-memory) -> ACK; on persist failure: no apply, no ACK
(a standby never ACKs an entry it didn't persist). Covers single + batch
catch-up; OPCode_replEntry has no separate apply path (allow-list only), so all
live WAL apply now persists. Dedup guard (lsn+1<=current_lsn) makes re-streams
idempotent while still ACKing.

TDD: standby write -> drop persist_t -> reopen + persist_recover replays it
(single, 3-entry crc chain, post-dedup); checkpoint/oversized rejected. Full
controller suite green under ASAN; persist_wal_async / recovery / truncation /
streamer / wire untouched.
This commit is contained in:
Rongsong Shen 2026-06-14 13:10:02 +00:00
parent 007bf0c403
commit 708f2279e3
4 changed files with 359 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

15
src/include/persist.h
View file

@ -116,6 +116,21 @@ int persist_wal_sync_append(persist_t* p, uint32_t type,
const void* payload, uint32_t len,
int* sync_rc_out);
/* Write a REPLICATED entry (received from the primary) into the local WAL
* at its GIVEN lsn — used by standbys to durably persist streamed entries
* BEFORE acking them (persist-before-ACK; an ACK must imply durability).
*
* Unlike persist_wal_async it does NOT assign a new lsn (the lsn is supplied
* by the primary) and does NOT repl_broadcast (standbys never re-replicate).
* It builds a SELF-consistent CRC chain (the streamed entry carries no crc),
* fsyncs the segment before returning, and is idempotent on re-stream: if the
* lsn is already persisted (lsn+1 <= current_lsn) it is a no-op returning 0.
*
* WAL_CHECKPOINT is rejected with -EINVAL (never streamed to standbys).
* Returns 0 on success (incl. dedup no-op), negative errno on failure. */
int persist_wal_write_replicated(persist_t* p, uint64_t lsn, uint32_t type,
const void* payload, uint32_t len);
/* ============================================================
* Write-path commit classification (ha-commit-barrier-rollout)
* ============================================================ */