fix bug-1060 bug-1067

This commit is contained in:
Rongsong Shen 2026-06-24 17:05:43 +08:00
parent 6067ba6c94
commit 61e1391cd0
28 changed files with 7021 additions and 2926 deletions

View file

@ -70,7 +70,7 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft
**Election Safety** (≤1 leader per term): Enforced by `votedFor` — a server grants at most one vote per term (`raft_on_vote_request` line 145-151). A candidate needs majority votes; each server votes at most once per term.
**Leader Completeness** (committed entries survive): Freshness gate in `RequestVote` — candidate must have `lastLogTerm ≥ voter` AND `lastLogIndex ≥ voter`. Since committed entries are on a majority, the new leader must have all committed entries. `persist_truncate_wal_above(safe_committed_lsn)` removes any divergent entries above the safe commit point.
**Leader Completeness** (committed entries survive): Freshness gate in `RequestVote` — candidate must have `lastLogTerm ≥ voter` AND `lastLogIndex ≥ voter`. Since committed entries are on a majority, the new leader must have all committed entries. The post-election truncation removes only a divergent tail **above the leader's own durable log**: the cutoff is `ha_promotion_truncate_floor(safe_committed_lsn, durable_tip) = max(safe_committed_lsn, persist_durable_floor)`, so `persist_truncate_wal_above` is a no-op on the leader's own durable entries. **A new leader MUST NEVER truncate its own log** — entries it holds durably were acked by it (as a standby) and so were committed by a prior leader; truncating to the bare majority element destroys them (bug-1060: granted `{self=47, peer=44}` → `safe=44` cut the leader's own committed 45..47 → divergent WALs + permanent prevote deadlock).
**State Machine Safety** (no divergent entries): Monotonic `committed_lsn` + truncation on promotion. Entries at or below `committed_lsn` are identical across all servers.
@ -88,7 +88,7 @@ After winning election, the new primary determines `safe_committed_lsn` from vot
safe_committed_lsn = max(lsn : count of voters with peerLastLogLsn >= lsn) + 1 >= majority)
```
The `RequestVoteReply` message already carries `peerConfigTag` and `peerLastLogLsn` from each voter. The winning candidate uses these (from `ha_elect_ctx_t.votes[]`) to compute `safe_committed_lsn` immediately in `raft_cb_become_leader`, then calls `persist_truncate_wal_above(safe_committed_lsn)`.
The `RequestVoteReply` message already carries `peerConfigTag` and `peerLastLogLsn` from each voter. The winning candidate uses these (from `ha_elect_ctx_t.votes[]`) to compute `safe_committed_lsn` immediately in `raft_cb_become_leader`, then truncates at `ha_promotion_truncate_floor(safe_committed_lsn, persist_durable_floor)` — i.e. `max(safe_committed_lsn, own durable tip)`, so it never truncates the leader's own durable log (bug-1060 Leader Completeness; `safe_committed_lsn` is still used as the commit lower-bound and the promotion-gate HWM).
**Never** use `last_known_committed_lsn` from the previous primary's ctrlhello for truncation — it may be stale (the old primary may have committed entries after the last ctrlhello was sent).
@ -100,8 +100,9 @@ When modifying HA protocol code (election, replication, heartbeat), verify:
- [ ] `commitIndex` never decreases (monotonic)
- [ ] A leader only commits entries from its current term (or equivalent via per-entry majority ack)
- [ ] `safe_committed_lsn` is derived from voter replies, not from stale ctrlhello
- [ ] WAL truncation on promotion only removes entries above the safe commit point
- [ ] WAL truncation on promotion never falls below the leader's OWN durable tip — cutoff = `max(safe_committed_lsn, persist_durable_floor)` (`ha_promotion_truncate_floor`); a new leader never truncates its own committed log (bug-1060)
- [ ] Freshness gate in RequestVote: candidate must have `lastLogTerm >= voter` AND `lastLogIndex >= voter`
- [ ] AppendEntries carries TWO distinct terms: the HEADER term = leader's CURRENT term (UpdateTerm/stale-check) and each ENTRY's STORED term (logOk/persist). For catch-up of historical entries they DIFFER — the AE header term must be the leader's current term (`wal_catchup_entry_t.leader_current_term`), never the entry's stored term, or the library stale-rejects every cross-term entry (bug-1067)
- [ ] `committed_lsn` advances only as a contiguous LSN-ordered prefix — no out-of-order or per-entry commit
- [ ] Durability flushing makes a contiguous prefix durable; `durable_lsn` is a prefix watermark (never a per-entry set)
- [ ] Causal dependencies are appended in order (cause before effect, e.g. submit before start)