lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): refuse replicated WAL writes before this boot's recovery (bug-1920)

Browse source 
A restarted standby accepts the primary's connection during its startup
delay, before persist_recover runs. Catch-up entries written into the
un-recovered persist_t opened seg-...0001 with O_TRUNC and rewrote it,
and after recovery the stale fd kept filing new LSNs under that segment
name. The primary's filename-ordered streamer then skipped them, so a
standby needing that range never caught up and commits stalled.

persist_wal_write_replicated now returns -EBUSY until recovery completes
(the bug-1416 recovery_complete gate). ha_apply_wal_catchup_entry and
ha_handle_snapshot_chunk drop pre-recovery traffic without ACK or NACK.
Catch-up streams from min(NACK-derived join position, standby report),
and the streamer warns when a non-empty range yields no entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-04 12:02:37 +00:00
parent f01bb7de34
commit 762c6f1883
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
6 changed files with 242 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

30
src/controller/ha.c
View file

@ -3658,6 +3658,21 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
if (!ha || !p || !entry)
return -EINVAL;
/* bug-1920: before this boot's persist_recover the local log is not
* loaded: persisting would corrupt the on-disk segment layout (see
* persist_wal_write_replicated), and a NACK would walk the primary's
* next_index back past entries this node really holds. Drop silently --
* no persist, no ACK, no NACK; the first state report after recovery
* carries the real position and catch-up resumes from there. */
if (!persist_recovery_complete(p)) {
static _Atomic int s_logged;
if (!atomic_exchange_explicit(&s_logged, 1, memory_order_relaxed))
INFO("ha_apply_wal_catchup_entry: replication received before "
"this boot's WAL recovery completed — dropping until "
"recovered (bug-1920)");
return 0;
}
/* ── ha-term-safety task 3.2: term-carrying AppendEntries → library route ──
* A term-safe AppendEntries (one sent by repl_send_append_entries / the
* library send path) carries a non-zero entry term stamp. Route it through
@ -3965,6 +3980,12 @@ int ha_handle_snapshot_chunk(ha_state_t *ha, persist_t *p,
if (!ha || !p || !snap_dir || !chunk)
return -EINVAL;
/* bug-1920: same pre-recovery drop as ha_apply_wal_catchup_entry -- a
* snapshot installed before persist_recover would be clobbered or mixed
* with the replay. The transfer restarts from chunk 0 after recovery. */
if (!persist_recovery_complete(p))
return 0;
char part_path[4096 + 16];
snprintf(part_path, sizeof(part_path), "%s/snap.part", snap_dir);
@ -4679,7 +4700,14 @@ int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
* corrected position at least once. */
if (c) {
uint64_t join_lsn;
if (repl_get_catchup_join_lsn(&c->persist.repl, ep, &join_lsn))
/* bug-1920: never above the standby's own report. The NACK backoff
* moves next_index down one per reject, so join_lsn can sit far above
* where the standby's log really ends; streaming from there is
* rejected (one step per heartbeat) or finds nothing to send. The
* report is an upper bound on what the standby holds; a lower
* join_lsn (a conflicted suffix the report cannot see) still wins. */
if (repl_get_catchup_join_lsn(&c->persist.repl, ep, &join_lsn)
&& join_lsn < standby_lsn)
standby_lsn = join_lsn;
}