forked from lei.chen/myxce
fix(ha-term-safety): bug-828 FIXED — remove premature pre-election WAL truncate (committed-job-loss on clean failover)
ROOT CAUSE (confirmed via live instrumentation — NOT a durability violation): ha_enter_real_election truncated the WAL to last_known_committed_lsn BEFORE the vote. That watermark lags the true commit point (the primary committed LSN 2-5 on a durable majority then died before a heartbeat carried committed=5 to the standby), so the cutoff was 1; persist_truncate_wal_above -> clamp_durable_lsn lowered the standby's durable_lsn 5->1, physically destroying durably-committed entries before the election even computed the safe point. The candidate then advertised floor=1, safe_committed_lsn=1, and the post-election truncate kept only LSN 1. This directly violated the documented rule: 'Never use last_known_committed _lsn for truncation — it may be stale.' FIX: delete the premature truncate. The candidate keeps its full durable log; the ONLY truncates are now the authoritative votes-derived post-election persist_truncate_wal_above(safe_committed_lsn) (raft_cb_become_leader) + the library on_truncate_from (term,lsn) reconcile — both before any commit under the new leader. SAFE (reviewer-verified): a divergent uncommitted tail can only exist at a stale/lower term (single-writer); the library's term-primary (lastLogTerm, lastLogIndex) freshness gate denies such a candidate — correct now that task 3.4 fixed the term axis (the prerequisite). No divergent entry can win-and-commit; no bug-339 election-loop. 177 unit tests + a RED-proven regression test. LIVE: clean failover now preserves committed jobs — walcompare committed-prefix identical across all 3 members, both jobs present on disk, across multiple runs. (Residual harness EXIT 1 is the SEPARATE pre-existing libxce client bug-854: client can't reach the live primary when the dead node is first in the masters list — a client-discovery defect, orthogonal to server-side data safety.)
This commit is contained in:
parent
ff869e4fc0
commit
b77d412256
2 changed files with 91 additions and 6 deletions
|
|
@ -5184,12 +5184,29 @@ static void ha_enter_real_election(ha_state_t *ha, cluster_t *c)
|
|||
return;
|
||||
}
|
||||
|
||||
/* Truncate uncommitted WAL entries above committed_lsn (mirrors the
|
||||
* existing failover sequence in ha_on_primary_ctrl_fd_rdhup). */
|
||||
uint64_t committed = atomic_load_explicit(&ha->last_known_committed_lsn,
|
||||
memory_order_acquire);
|
||||
if (committed > 0)
|
||||
persist_truncate_wal_above(&c->persist, committed);
|
||||
/* bug-828 final slice (Leader Completeness): do NOT pre-truncate the WAL
|
||||
* here. The earlier code truncated above ha->last_known_committed_lsn
|
||||
* BEFORE the vote — but that watermark is the LOCALLY-LEARNED commit point,
|
||||
* which LAGS the true cluster commit point (the primary commits an entry on
|
||||
* a durable majority, then dies before a heartbeat carries that committed
|
||||
* LSN to this standby). persist_truncate_wal_above clamps durable_lsn down
|
||||
* to the cutoff, so a standby that had DURABLY persisted committed entries
|
||||
* [committed+1 .. tip] would discard them and then advertise the stale-low
|
||||
* cutoff as its durable floor. Every voter doing the same made
|
||||
* safe-committed-from-votes collapse to the stale watermark, and the
|
||||
* AUTHORITATIVE post-election truncate (raft_cb_become_leader,
|
||||
* persist_truncate_wal_above(safe_committed_lsn)) then kept only that stale
|
||||
* prefix — losing committed jobs on a clean failover (bug-828/795).
|
||||
*
|
||||
* Raft model: a candidate does NOT truncate its log before campaigning. It
|
||||
* keeps its full durable log and advertises its real durable tip
|
||||
* (ha_candidate_advertise_lsn == persist_durable_floor), so voters holding
|
||||
* more committed entries deny a colder candidate and the winner has every
|
||||
* committed entry (Leader Completeness). Divergent UNCOMMITTED tails are
|
||||
* removed ONLY by the authoritative post-election safe-commit truncate
|
||||
* (above the majority-held point) and by the library's per-entry (term,lsn)
|
||||
* reconcile (on_truncate_from) during subsequent AppendEntries — never by a
|
||||
* stale-watermark pre-truncate that can erase committed state. */
|
||||
|
||||
/* Candidate freshness for the RequestVote payload (ha-commit-durability
|
||||
* task 4.2). The TAG axis uses the same config-epoch fold the voter side
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue