fix(ha): bug-1421 close term-desync AE-reject blindspot in HA election

A primary never learns of a follower's higher term via AppendEntries-reject
responses: ha_raft_on_repl_ack only routes AE replies through
raft_on_append_entries_reply for success=1 ACKs, and that function itself
drops any reply whose term != s->term (raft.c stale-term drop) before it
could adopt a higher term anyway. Net effect: once a follower outpaces a
stale primary's term, every AppendEntries the primary sends is rejected
forever and it can never regain write-quorum — a permanent election livelock
(surfaced as "no primary elected at term>8" in HA soak testing).

Adds two fallback term-discovery vectors that fire regardless of AE-reject
outcome:
- StateReport heartbeat-reply now carries the standby's current term
  (controller.capnp, typemap.h) so the primary learns a higher term on every
  convergence cycle.
- New raft_on_higher_term_observed() mirrors the UpdateTerm step (strictly
  greater-than only; touches term/voted_for/role; persists before notifying
  on_become_follower) and is wired into both the heartbeat state-report path
  and the WAL-NACK path in repl_ack_decode_and_dispatch.

Validated: 265 controller + 287 lib unit tests pass; 10-round sequential and
chaotic HA soaks both PASS with zero job loss and full leadership coverage.
This commit is contained in:
Rongsong Shen 2026-07-09 11:56:46 +00:00
parent 0941cf9649
commit 327dbb9bb3
5 changed files with 62 additions and 1 deletions

View file

@ -193,10 +193,14 @@ typedef struct {
uint64_t peer_last_log_lsn;
} request_vote_reply_t;
/* Heartbeat reply state report — carries standby's applied_lsn and config_tag */
/* Heartbeat reply state report — carries standby's applied_lsn and config_tag,
* plus its raft term so the primary can discover a higher term it lost track
* of (bug-1421 term-discovery: an AE reject response never reaches the raft
* library, so a heartbeat-reply term stamp is the fallback path). */
typedef struct {
uint64_t applied_lsn;
uint64_t config_tag;
uint64_t term;
} state_report_t;
/* WAL apply ACK — sent by standby to primary after applying a WAL catch-up