fix(ha-term-safety): bug-828 FIXED — remove premature pre-election WAL truncate (committed-job-loss on clean failover)

ROOT CAUSE (confirmed via live instrumentation — NOT a durability violation):
ha_enter_real_election truncated the WAL to last_known_committed_lsn BEFORE the
vote. That watermark lags the true commit point (the primary committed LSN 2-5 on
a durable majority then died before a heartbeat carried committed=5 to the
standby), so the cutoff was 1; persist_truncate_wal_above -> clamp_durable_lsn
lowered the standby's durable_lsn 5->1, physically destroying durably-committed
entries before the election even computed the safe point. The candidate then
advertised floor=1, safe_committed_lsn=1, and the post-election truncate kept only
LSN 1. This directly violated the documented rule: 'Never use last_known_committed
_lsn for truncation — it may be stale.'

FIX: delete the premature truncate. The candidate keeps its full durable log; the
ONLY truncates are now the authoritative votes-derived post-election
persist_truncate_wal_above(safe_committed_lsn) (raft_cb_become_leader) + the
library on_truncate_from (term,lsn) reconcile — both before any commit under the
new leader.

SAFE (reviewer-verified): a divergent uncommitted tail can only exist at a
stale/lower term (single-writer); the library's term-primary (lastLogTerm,
lastLogIndex) freshness gate denies such a candidate — correct now that task 3.4
fixed the term axis (the prerequisite). No divergent entry can win-and-commit; no
bug-339 election-loop. 177 unit tests + a RED-proven regression test.

LIVE: clean failover now preserves committed jobs — walcompare committed-prefix
identical across all 3 members, both jobs present on disk, across multiple runs.
(Residual harness EXIT 1 is the SEPARATE pre-existing libxce client bug-854:
client can't reach the live primary when the dead node is first in the masters
list — a client-discovery defect, orthogonal to server-side data safety.)
This commit is contained in:
Rongsong Shen 2026-06-23 04:06:18 +00:00
parent ff869e4fc0
commit b77d412256
2 changed files with 91 additions and 6 deletions

View file

@ -5184,12 +5184,29 @@ static void ha_enter_real_election(ha_state_t *ha, cluster_t *c)
return;
}
/* Truncate uncommitted WAL entries above committed_lsn (mirrors the
* existing failover sequence in ha_on_primary_ctrl_fd_rdhup). */
uint64_t committed = atomic_load_explicit(&ha->last_known_committed_lsn,
memory_order_acquire);
if (committed > 0)
persist_truncate_wal_above(&c->persist, committed);
/* bug-828 final slice (Leader Completeness): do NOT pre-truncate the WAL
* here. The earlier code truncated above ha->last_known_committed_lsn
* BEFORE the vote — but that watermark is the LOCALLY-LEARNED commit point,
* which LAGS the true cluster commit point (the primary commits an entry on
* a durable majority, then dies before a heartbeat carries that committed
* LSN to this standby). persist_truncate_wal_above clamps durable_lsn down
* to the cutoff, so a standby that had DURABLY persisted committed entries
* [committed+1 .. tip] would discard them and then advertise the stale-low
* cutoff as its durable floor. Every voter doing the same made
* safe-committed-from-votes collapse to the stale watermark, and the
* AUTHORITATIVE post-election truncate (raft_cb_become_leader,
* persist_truncate_wal_above(safe_committed_lsn)) then kept only that stale
* prefix — losing committed jobs on a clean failover (bug-828/795).
*
* Raft model: a candidate does NOT truncate its log before campaigning. It
* keeps its full durable log and advertises its real durable tip
* (ha_candidate_advertise_lsn == persist_durable_floor), so voters holding
* more committed entries deny a colder candidate and the winner has every
* committed entry (Leader Completeness). Divergent UNCOMMITTED tails are
* removed ONLY by the authoritative post-election safe-commit truncate
* (above the majority-held point) and by the library's per-entry (term,lsn)
* reconcile (on_truncate_from) during subsequent AppendEntries — never by a
* stale-watermark pre-truncate that can erase committed state. */
/* Candidate freshness for the RequestVote payload (ha-commit-durability
* task 4.2). The TAG axis uses the same config-epoch fold the voter side