wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(ha-wal-replication): standby durably persists replicated WAL before ACK

Browse source 
A standby applied replicated WAL entries via apply_wal_entry (the replay/
in-memory path) which restores entities + advances applied_lsn but NEVER writes
the entry to the standby's WAL segment. The standby then ACKed (feeding the
primary's commit majority) without persisting — a Raft durability violation:
entries counted toward a commit could be lost on standby restart, and standby
WAL files stayed empty (contradicting steering: log[i] = WAL segments for every
node). This is why ctrl-2/ctrl-3 received + acked the streamed own-term NOOP yet
kept empty seg-*.wal.

Fix: new persist_wal_write_replicated(p, lsn, type, payload, len) writes the
entry to the local WAL at the PRIMARY's lsn (no lsn-reassignment), with a
self-consistent intra-segment CRC chain (the streamed entry carries no term/crc;
each node chains independently and recovery validates intra-segment), fdatasyncs
for durability, advances current_lsn = max(current_lsn, lsn+1), rotates segments,
and does NOT re-broadcast. Wired into ha_apply_single_entry as
persist(durable) -> apply(in-memory) -> ACK; on persist failure: no apply, no ACK
(a standby never ACKs an entry it didn't persist). Covers single + batch
catch-up; OPCode_replEntry has no separate apply path (allow-list only), so all
live WAL apply now persists. Dedup guard (lsn+1<=current_lsn) makes re-streams
idempotent while still ACKing.

TDD: standby write -> drop persist_t -> reopen + persist_recover replays it
(single, 3-entry crc chain, post-dedup); checkpoint/oversized rejected. Full
controller suite green under ASAN; persist_wal_async / recovery / truncation /
streamer / wire untouched.
This commit is contained in:
Rongsong Shen 2026-06-14 13:10:02 +00:00
parent 007bf0c403
commit 708f2279e3
4 changed files with 359 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

21
src/controller/ha.c
View file

@ -1030,6 +1030,27 @@ static void ha_advance_applied_lsn(ha_state_t *ha, uint64_t new_lsn)
* Returns 0 on success, non-zero on error (caller logs and skips). */
static int ha_apply_single_entry(persist_t *p, wal_catchup_entry_t *e)
{
/* Persist-before-apply-before-ACK: durably write the replicated entry to
* this standby's own WAL BEFORE applying it in memory (and before the
* caller ACKs). The ACK feeds the primary's commit majority, so it must
* imply durability — a standby must not ACK an entry it could lose on
* restart (Raft log[i] includes standbys; ha-async-io.md).
*
* persist_wal_write_replicated does NOT assign a new lsn (uses e->lsn) and
* does NOT re-broadcast; it fsyncs before returning and is idempotent on
* re-stream. WAL_CHECKPOINT is never streamed, but it would be rejected
* by both this primitive and apply_wal_entry. */
int prc = persist_wal_write_replicated(p, e->lsn, (uint32_t)e->entry_type,
e->raw_payload,
(uint32_t)e->raw_payload_len);
if (prc != 0) {
ERROR("ha_apply_single_entry: persist_wal_write_replicated failed for "
"lsn=%" PRIu64 " type=%u rc=%d — not applying/acking",
e->lsn, (unsigned)e->entry_type, prc);
/* No apply, no ACK on persist failure (mirrors no-ACK-on-apply-fail). */
return prc;
}
wal_entry_hdr_t hdr;
memset(&hdr, 0, sizeof(hdr));
hdr.lsn = e->lsn;