wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(ha): bug-1421 close term-desync AE-reject blindspot in HA election

Browse source 
A primary never learns of a follower's higher term via AppendEntries-reject
responses: ha_raft_on_repl_ack only routes AE replies through
raft_on_append_entries_reply for success=1 ACKs, and that function itself
drops any reply whose term != s->term (raft.c stale-term drop) before it
could adopt a higher term anyway. Net effect: once a follower outpaces a
stale primary's term, every AppendEntries the primary sends is rejected
forever and it can never regain write-quorum — a permanent election livelock
(surfaced as "no primary elected at term>8" in HA soak testing).

Adds two fallback term-discovery vectors that fire regardless of AE-reject
outcome:
- StateReport heartbeat-reply now carries the standby's current term
  (controller.capnp, typemap.h) so the primary learns a higher term on every
  convergence cycle.
- New raft_on_higher_term_observed() mirrors the UpdateTerm step (strictly
  greater-than only; touches term/voted_for/role; persists before notifying
  on_become_follower) and is wired into both the heartbeat state-report path
  and the WAL-NACK path in repl_ack_decode_and_dispatch.

Validated: 265 controller + 287 lib unit tests pass; 10-round sequential and
chaotic HA soaks both PASS with zero job loss and full leadership coverage.
This commit is contained in:
Rongsong Shen 2026-07-09 11:56:46 +00:00
parent 0941cf9649
commit 327dbb9bb3
5 changed files with 62 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

19
src/controller/ha.c
View file

@ -2818,6 +2818,7 @@ void ha_nop_reply_state_report(ha_state_t *ha, myconf_t *myconf,
sr->applied_lsn = atomic_load_explicit(&ha->applied_lsn,
memory_order_acquire);
sr->config_tag = myconf ? myconf->config_version_tag : 0;
sr->term = atomic_load_explicit(&ha->term, memory_order_acquire);
reply_out->data_which = RequestReply_data_stateReportMsg;
reply_out->data.state_report = sr;
}
@ -5487,8 +5488,21 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx,
state_report_t sr;
sr.applied_lsn = (uint64_t)sr_raw.appliedLsn;
sr.config_tag = (uint64_t)sr_raw.configTag;
sr.term = (uint64_t)sr_raw.term;
capn_free(&capn_ctx);
/* bug-1421: a primary never learns of a follower's higher term via
* the AE-response path (an AE reject is dropped before it reaches
* raft_on_append_entries_reply — see ha_raft_on_repl_ack). The
* heartbeat-reply state report is the fallback term-discovery
* vector: it fires on every convergence cycle regardless of
* replication outcome, so a stale-term primary steps down here
* instead of retrying AppendEntries against a follower that will
* reject every entry forever. raft_on_higher_term_observed only
* touches (term, voted_for, role) — never peer-indexed
* bookkeeping — so it is safe to call before peer_idx is resolved. */
(void)raft_on_higher_term_observed(&ha->raft, sr.term);
/* peer_idx from standby_ep_ctx_t (layout-compatible with vote_ep_ctx_t) */
int peer_idx = vctx->peer_idx;
uint64_t committed_lsn = 0;
@ -5580,6 +5594,11 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx,
* deferred continuation task. */
ha_walack_on_ack(c, ep, vctx->peer_idx, acked_lsn);
} else if (!success) {
/* bug-1421: a WAL NACK can also carry a higher term (the
* standby rejected this entry as stale) — route it through
* the same term-discovery path as the heartbeat state report
* so the primary steps down instead of retrying forever. */
(void)raft_on_higher_term_observed(&ha->raft, reply_term);
WARN("repl_ack_decode_and_dispatch: WAL NACK from standby"
" lsn=%lu success=0 — ignoring", (unsigned long)acked_lsn);
}