wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(ha-term-safety): remove task-2.4 commit min-ceiling (Option A) + fix bug-991 catch-up off-by-one

Browse source 
Two coupled fixes for a live commit-stall (committed_lsn pinned at the own-term
NOOP while a 3-node cluster was durable to LSN 5 — jobs acked but never committed):

bug-991: the WAL catch-up applied_lsn was advanced to lsn+1 (next-to-apply) in
the three live/catch-up apply paths, but recovery/checkpoint/init and the leader's
catch-up streamer treat it as last-held. The over-advertise made the streamer skip
an LSN -> library logOk wedge -> no WalAck. Fixed to last-held (= lsn).

Option A (design reconsideration of REVIEW ISSUE 3): removed the
committed = min(durable-majority, raft_commit_index) ceiling. Under single-writer
the durable-majority watermark is already (term,lsn)-current-term-safe (term-carrying
replication + (term,lsn) reconcile/truncate-before-durable + own-term NOOP gate +
post-election truncate-to-safe), so the library commit_index was a redundant,
starvation-prone gate that pinned committed below the safe durable majority whenever
the library lagged. The durable-majority barrier + own-term writable gate are now
authoritative; raft_commit_index is inert-advisory.

Safety: reviewer-verified the durable-majority watermark is (term,lsn)-safe and
Figure-8 holds via the own-term gate (new RED->GREEN test); bug-851 cannot recur.
Live: commit trace 4->5 + walcompare committed-prefix identical across 3 nodes.
177 unit tests pass.
This commit is contained in:
Rongsong Shen 2026-06-23 00:21:06 +00:00
parent df4bc4f86a
commit 83fbed2370
5 changed files with 396 additions and 172 deletions
Download patch file Download diff file Expand all files Collapse all files

42
src/controller/ha.c
View file

@ -1966,18 +1966,26 @@ void ha_fini(ha_state_t *ha)
* ha_apply_wal_catchup_entry: apply WAL catch-up entries received from the
* primary to local persistent storage and advance applied_lsn on success.
*
* applied_lsn semantic (bug-991): the LAST CONTIGUOUSLY-HELD LSN — the same
* meaning recovery (= durable_floor), checkpoint (= checkpoint_lsn), and init
* (= 0) assign it, and the watermark the standby advertises to the leader
* (sr->applied_lsn) so the leader's catch-up streamer streams the half-open
* range (applied_lsn, committed]. NEVER lsn+1 (next-to-apply): a +1 makes the
* standby advertise one LSN too high, so the leader skips the real next-needed
* entry and the standby's library logOk wedges forever (no WalAck → commit stall).
*
* Two modes (Req 5.3, 5.5):
*
* Batch mode (entry->n_entries > 0):
* Iterate entry->entries[0..n_entries-1] in order. Apply each sub-entry
* individually via apply_wal_entry(). NULL sub-entries are treated as
* corrupt and skipped (Req 3.3). After ALL sub-entries are processed,
* advance applied_lsn to last-successfully-applied lsn+1 so the next
* advance applied_lsn to the last-successfully-applied lsn so the next
* heartbeat reply reflects post-batch state (Req 5.3).
*
* Single-entry mode (entry->n_entries == 0):
* Backward-compatible path from task 2.1. Apply entry->lsn/entry_type
* directly and advance applied_lsn to lsn+1 on success.
* directly and advance applied_lsn to lsn on success.
*
* In both modes:
* - applied_lsn is advanced atomically and monotonically.
@ -2110,9 +2118,17 @@ static int ha_apply_term_entry_via_library(ha_state_t *ha, persist_t *p,
uint64_t after_term = persist_entry_term(p, e->lsn);
if (after_term == e->term) {
/* Accepted (appended / conflict-truncate-then-persist) or already-have
* (idempotent re-stream): ACK + advance applied on that signal. */
* (idempotent re-stream): ACK + advance applied on that signal.
*
* bug-991: applied_lsn is the "last contiguously-held LSN" (the same
* semantic recovery/checkpoint/init use, and the watermark the leader's
* catch-up streamer reads via sr->applied_lsn to stream (standby_lsn,
* committed]). Advance to e->lsn (the LSN we just held), NOT e->lsn + 1:
* a +1 over-advertises one LSN, so the leader's next stream begins at
* e->lsn + 2 and SKIPS the real next-needed entry e->lsn + 1, wedging
* logOk and stalling commit. */
(void)before_term;
ha_advance_applied_lsn(ha, e->lsn + 1);
ha_advance_applied_lsn(ha, e->lsn);
ha_send_wal_ack_reply(ha, ep, c, e->lsn);
return 1;
}
@ -2199,10 +2215,13 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
had_good = 1;
}
/* Advance applied_lsn to last_good_lsn+1 after the full batch.
* Only advance if at least one entry was applied successfully. */
/* Advance applied_lsn to last_good_lsn after the full batch (bug-991:
* last contiguously-held LSN, NOT last_good_lsn + 1 — see the
* ha_apply_term_entry_via_library comment for why the +1 wedged the
* leader's catch-up streamer). Only advance if at least one entry was
* applied successfully. */
if (had_good)
ha_advance_applied_lsn(ha, last_good_lsn + 1);
ha_advance_applied_lsn(ha, last_good_lsn);
return 0;
}
@ -2220,9 +2239,12 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
return rc;
}
/* Advance applied_lsn atomically (lsn+1 mirrors persist->current_lsn
* after a successful apply). */
ha_advance_applied_lsn(ha, entry->lsn + 1);
/* Advance applied_lsn atomically to the last contiguously-held LSN
* (bug-991: entry->lsn, NOT entry->lsn + 1 — see the
* ha_apply_term_entry_via_library comment for the streamer-skip wedge a +1
* caused). This is the same last-held semantic recovery/checkpoint/init use,
* and matches the (standby_lsn, committed] range the leader streams. */
ha_advance_applied_lsn(ha, entry->lsn);
/* Task 3.1: ACK the successfully applied single entry */
ha_send_wal_ack_reply(ha, ep, c, entry->lsn);