fix(ha): bug-1421 close term-desync AE-reject blindspot in HA election

A primary never learns of a follower's higher term via AppendEntries-reject
responses: ha_raft_on_repl_ack only routes AE replies through
raft_on_append_entries_reply for success=1 ACKs, and that function itself
drops any reply whose term != s->term (raft.c stale-term drop) before it
could adopt a higher term anyway. Net effect: once a follower outpaces a
stale primary's term, every AppendEntries the primary sends is rejected
forever and it can never regain write-quorum — a permanent election livelock
(surfaced as "no primary elected at term>8" in HA soak testing).

Adds two fallback term-discovery vectors that fire regardless of AE-reject
outcome:
- StateReport heartbeat-reply now carries the standby's current term
  (controller.capnp, typemap.h) so the primary learns a higher term on every
  convergence cycle.
- New raft_on_higher_term_observed() mirrors the UpdateTerm step (strictly
  greater-than only; touches term/voted_for/role; persists before notifying
  on_become_follower) and is wired into both the heartbeat state-report path
  and the WAL-NACK path in repl_ack_decode_and_dispatch.

Validated: 265 controller + 287 lib unit tests pass; 10-round sequential and
chaotic HA soaks both PASS with zero job loss and full leadership coverage.
This commit is contained in:
Rongsong Shen 2026-07-09 11:56:46 +00:00
parent 0941cf9649
commit 327dbb9bb3
5 changed files with 62 additions and 1 deletions

View file

@ -851,3 +851,24 @@ int raft_on_append_entries_reply(raft_state_t *s, int peer_idx, int success,
return RAFT_OK;
}
int raft_on_higher_term_observed(raft_state_t *s, uint64_t observed_term)
{
if (observed_term <= s->term)
return RAFT_OK;
int was_leader = (s->role == RAFT_LEADER);
s->term = observed_term;
s->voted_for = -1;
s->voted_for_term = 0;
s->role = RAFT_FOLLOWER;
/* Persist the adopted term before notifying (mirrors UpdateTerm ordering
* in raft_on_append_entries_request, Req 6.2). */
s->cbs->on_persist(s->ctx, s->term, -1);
if (was_leader)
s->cbs->on_become_follower(s->ctx, s->term, -1);
if (s->cbs->on_trace)
s->cbs->on_trace(s->ctx, "term_adopt_observed", NULL);
return RAFT_BECAME_FOLLOWER;
}

View file

@ -403,6 +403,22 @@ int raft_on_append_entries_request(raft_state_t *s, int leader_idx,
int raft_on_append_entries_reply(raft_state_t *s, int peer_idx, int success,
uint64_t match_index, uint64_t reply_term);
/*
* raft_on_higher_term_observed: adopt a higher term learned from ANY
* peer-carried term stamp that is not itself a full RequestVote/AppendEntries
* exchange (e.g. a heartbeat-reply state report) — closes bug-1421, where a
* primary never learns of a follower's higher term because AE reject
* responses are not routed to raft_on_append_entries_reply (which drops any
* reply whose term != s->term, so it cannot be reused here without corrupting
* match_index/next_index on the common equal-term case). Mirrors the
* UpdateTerm step shared by raft_on_append_entries_request (raft.tla 406):
* strictly-greater-than only; touches ONLY (term, voted_for, role) — never
* peer-indexed bookkeeping — so it is safe to call with an unresolved/unknown
* peer_idx. No-op (RAFT_OK) when observed_term <= s->term. Returns
* RAFT_BECAME_FOLLOWER when it steps a leader down.
*/
int raft_on_higher_term_observed(raft_state_t *s, uint64_t observed_term);
/*
* raft_advance_commit: leader recomputes its commit index (raft.tla
* AdvanceCommitIndex, 259). A SEPARATE action from raft_on_append_entries_reply