fix(controller): refuse replicated WAL writes before this boot's recovery (bug-1920)

A restarted standby accepts the primary's connection during its startup
delay, before persist_recover runs. Catch-up entries written into the
un-recovered persist_t opened seg-...0001 with O_TRUNC and rewrote it,
and after recovery the stale fd kept filing new LSNs under that segment
name. The primary's filename-ordered streamer then skipped them, so a
standby needing that range never caught up and commits stalled.

persist_wal_write_replicated now returns -EBUSY until recovery completes
(the bug-1416 recovery_complete gate). ha_apply_wal_catchup_entry and
ha_handle_snapshot_chunk drop pre-recovery traffic without ACK or NACK.
Catch-up streams from min(NACK-derived join position, standby report),
and the streamer warns when a non-empty range yields no entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-04 12:02:37 +00:00
parent f01bb7de34
commit 762c6f1883
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
6 changed files with 242 additions and 2 deletions

View file

@ -2511,6 +2511,7 @@ int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
wal_catchup_entry_t *batch_entries[WAL_STREAM_BATCH_ENTRIES];
int batch_n = 0;
size_t batch_bytes = 0;
uint64_t n_queued = 0; /* entries put in a batch (bug-1920 diagnostic) */
/* ha-term-safety task 3.4: track the previous streamed entry's
* (lsn, term) so each catch-up entry carries the AppendEntries prev tuple
@ -2699,6 +2700,7 @@ int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
batch_entries[batch_n] = entry;
batch_n++;
n_queued++;
batch_bytes += sizeof(wal_entry_hdr_t) + hdr.payload_len;
/* Flush when batch is full (by count or accumulated size) */
@ -2741,8 +2743,21 @@ int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
}
}
/* bug-1920: a non-empty range that yields nothing means the on-disk
* segment layout does not match the filename-ordered scan above (an LSN
* filed under the wrong segment name) -- say so instead of logging a
* success the standby never sees. */
if (n_queued == 0) {
WARN("ha_stream_wal_to_standby: found NO entries for [%" PRIu64
"+1..%" PRIu64 "] in %d segment(s) under %s — nothing streamed "
"(WAL segment layout does not match LSN order?)",
standby_lsn, committed_lsn, nseg, wal_path);
return 0;
}
INFO("ha_stream_wal_to_standby: streamed WAL [%" PRIu64 "+1..%" PRIu64
"] to standby", standby_lsn, committed_lsn);
"] to standby (%" PRIu64 " entries)", standby_lsn, committed_lsn,
n_queued);
return 0;
}