fix(ha): bug-1421 close term-desync AE-reject blindspot in HA election
A primary never learns of a follower's higher term via AppendEntries-reject responses: ha_raft_on_repl_ack only routes AE replies through raft_on_append_entries_reply for success=1 ACKs, and that function itself drops any reply whose term != s->term (raft.c stale-term drop) before it could adopt a higher term anyway. Net effect: once a follower outpaces a stale primary's term, every AppendEntries the primary sends is rejected forever and it can never regain write-quorum — a permanent election livelock (surfaced as "no primary elected at term>8" in HA soak testing). Adds two fallback term-discovery vectors that fire regardless of AE-reject outcome: - StateReport heartbeat-reply now carries the standby's current term (controller.capnp, typemap.h) so the primary learns a higher term on every convergence cycle. - New raft_on_higher_term_observed() mirrors the UpdateTerm step (strictly greater-than only; touches term/voted_for/role; persists before notifying on_become_follower) and is wired into both the heartbeat state-report path and the WAL-NACK path in repl_ack_decode_and_dispatch. Validated: 265 controller + 287 lib unit tests pass; 10-round sequential and chaotic HA soaks both PASS with zero job loss and full leadership coverage.
This commit is contained in:
parent
0941cf9649
commit
327dbb9bb3
5 changed files with 62 additions and 1 deletions
|
|
@ -851,3 +851,24 @@ int raft_on_append_entries_reply(raft_state_t *s, int peer_idx, int success,
|
|||
|
||||
return RAFT_OK;
|
||||
}
|
||||
|
||||
int raft_on_higher_term_observed(raft_state_t *s, uint64_t observed_term)
|
||||
{
|
||||
if (observed_term <= s->term)
|
||||
return RAFT_OK;
|
||||
|
||||
int was_leader = (s->role == RAFT_LEADER);
|
||||
s->term = observed_term;
|
||||
s->voted_for = -1;
|
||||
s->voted_for_term = 0;
|
||||
s->role = RAFT_FOLLOWER;
|
||||
/* Persist the adopted term before notifying (mirrors UpdateTerm ordering
|
||||
* in raft_on_append_entries_request, Req 6.2). */
|
||||
s->cbs->on_persist(s->ctx, s->term, -1);
|
||||
if (was_leader)
|
||||
s->cbs->on_become_follower(s->ctx, s->term, -1);
|
||||
if (s->cbs->on_trace)
|
||||
s->cbs->on_trace(s->ctx, "term_adopt_observed", NULL);
|
||||
|
||||
return RAFT_BECAME_FOLLOWER;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -403,6 +403,22 @@ int raft_on_append_entries_request(raft_state_t *s, int leader_idx,
|
|||
int raft_on_append_entries_reply(raft_state_t *s, int peer_idx, int success,
|
||||
uint64_t match_index, uint64_t reply_term);
|
||||
|
||||
/*
|
||||
* raft_on_higher_term_observed: adopt a higher term learned from ANY
|
||||
* peer-carried term stamp that is not itself a full RequestVote/AppendEntries
|
||||
* exchange (e.g. a heartbeat-reply state report) — closes bug-1421, where a
|
||||
* primary never learns of a follower's higher term because AE reject
|
||||
* responses are not routed to raft_on_append_entries_reply (which drops any
|
||||
* reply whose term != s->term, so it cannot be reused here without corrupting
|
||||
* match_index/next_index on the common equal-term case). Mirrors the
|
||||
* UpdateTerm step shared by raft_on_append_entries_request (raft.tla 406):
|
||||
* strictly-greater-than only; touches ONLY (term, voted_for, role) — never
|
||||
* peer-indexed bookkeeping — so it is safe to call with an unresolved/unknown
|
||||
* peer_idx. No-op (RAFT_OK) when observed_term <= s->term. Returns
|
||||
* RAFT_BECAME_FOLLOWER when it steps a leader down.
|
||||
*/
|
||||
int raft_on_higher_term_observed(raft_state_t *s, uint64_t observed_term);
|
||||
|
||||
/*
|
||||
* raft_advance_commit: leader recomputes its commit index (raft.tla
|
||||
* AdvanceCommitIndex, 259). A SEPARATE action from raft_on_append_entries_reply
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue