lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(ha-term-safety): drive ACK path through raft_on_append_entries_reply + advance_commit (task 3.3)

Browse source 
Includes fix for bug-960: endpoint-type-aware peer-index resolution
(promoted-vote relative vs heartbeat-standby absolute) with fail-loud
bounds check, preventing an out-of-bounds match_index[] write / wrong-peer
false-quorum on the replication ACK path.
This commit is contained in:
Rongsong Shen 2026-06-22 09:25:44 +00:00
parent 754e586f8f
commit f1783f7952
4 changed files with 466 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

102
src/controller/ha.c
View file

@ -2894,6 +2894,86 @@ int ha_primary_log_append(cluster_t *c, uint32_t type,
return rc;
}
/* ha_repl_ack_abs_peer_idx (ha-term-safety task 3.3 remediation): resolve the
* ABSOLUTE library peer index from an ACK endpoint's stored peer_idx, branching
* on endpoint type EXACTLY as the standby_endpoints scan at ha.c:6187-6189.
*
* Two endpoint types reach the walAck branch of repl_ack_decode_and_dispatch,
* and their stored peer_idx has DIFFERENT meaning:
* - Promoted vote ep (vote_ep_ctx_t, ep_is_vote != 0): peer_idx is a
* self-EXCLUDED "votes" index (relative). Map it back to the absolute index
* — the inverse of raft_cb_send_vote_request's votes_idx remap (self_idx ==
* my_priority): abs = votes_idx < self_idx ? votes_idx : votes_idx + 1.
* - Heartbeat standby ep (standby_ep_ctx_t, ctx==NULL, ep_is_vote == 0):
* peer_idx is ALREADY the absolute candidate index — pass it through.
*
* The earlier unconditional remap corrupted the heartbeat path: a standby ep's
* already-absolute index was remapped a second time, crediting the WRONG peer
* (false quorum / stuck commit) or, near the array bound, an OUT-OF-BOUNDS
* match_index[] write. This resolver is the single source of truth so the
* production call site and the unit test exercise identical logic. */
int ha_repl_ack_abs_peer_idx(ha_state_t *ha, int ep_is_vote, int peer_idx)
{
int self_idx = ha->my_priority; /* self_idx == my_priority by convention */
if (ep_is_vote)
return (peer_idx < self_idx) ? peer_idx : peer_idx + 1;
return peer_idx; /* heartbeat standby ep: already absolute */
}
/* ha-term-safety task 3.3 (R4.1/R4.5): drive the raft library on a replication
* ACK. Route the reply through raft_on_append_entries_reply (updating
* match_index[]/next_index[]) then raft_advance_commit, so the library
* recomputes the current-term-gated commit (Figure-8) and MAY fire
* on_commit_advanced — the task-2.4 callback feeding the synchronous barrier.
*
* Peer-index contract: `abs_peer_idx` is the ABSOLUTE library peer index
* (j != self_idx), the same index space the library uses for
* match_index[]/next_index[]. The CALLER resolves it per endpoint type via
* ha_repl_ack_abs_peer_idx (promoted-vote eps carry a relative votes index;
* heartbeat standby eps carry an already-absolute index) — this seam must NOT
* remap again or it would corrupt the heartbeat path.
*
* Defense-in-depth: a peer index outside [0, raft.n_peers) (and never self) is
* rejected loudly rather than passed into raft_on_append_entries_reply, whose
* match_index[] write is unchecked. This fences an OOB / wrong-peer write at the
* controller boundary (no assert inside src/lib/raft/) and self-heals — the
* caller simply skips this ACK's library accounting.
*
* The library drops a stale-term reply (reply_term != s->term) inside
* raft_on_append_entries_reply, so a stale ACK never touches match_index[] and
* never advances commit. raft_advance_commit is leader-only (-EPERM otherwise);
* on a non-leader the call is a harmless no-op. Neither library call performs
* I/O — safe on the iomgr ACK path. */
int ha_raft_on_repl_ack(cluster_t *c, int abs_peer_idx, uint64_t acked_lsn,
uint64_t reply_term)
{
if (!c)
return -EINVAL;
/* Defense-in-depth bounds/self check: the library's match_index[] write is
* unchecked, so reject a bogus index here (fail loud, never corrupt). */
int self_idx = c->ha.my_priority;
int n_peers = c->ha.raft.n_peers;
if (abs_peer_idx < 0 || abs_peer_idx >= n_peers || abs_peer_idx == self_idx) {
ERROR("ha_raft_on_repl_ack: rejecting out-of-range peer index %d "
"(n_peers=%d self_idx=%d) — skipping library ACK accounting",
abs_peer_idx, n_peers, self_idx);
return -EINVAL;
}
/* A successful WalAck reports the standby's matched index (acked_lsn). The
* library applies it only when reply_term == s->term (stale-term drop). */
(void)raft_on_append_entries_reply(&c->ha.raft, abs_peer_idx, /*success*/1,
acked_lsn, reply_term);
/* Recompute the current-term-gated commit; fires on_commit_advanced exactly
* once on a strict advance of a current-term entry (else holds). Leader-only:
* -EPERM off the leader leaves commit_index unchanged with no callback. */
(void)raft_advance_commit(&c->ha.raft);
return 0;
}
/*
* ha_arm_own_term_gate: append the own-term WAL_NOOP and arm the commit
* engine's writable_lsn gate (ha-single-leader-role task 2.2, Req 1.4/2.1/2.2).
@ -4217,7 +4297,8 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx)
struct WalAckReply war_raw;
memset(&war_raw, 0, sizeof(war_raw));
read_WalAckReply(&war_raw, rep_s.data.walackreplymsg);
uint64_t acked_lsn = (uint64_t)war_raw.lsn;
uint64_t acked_lsn = (uint64_t)war_raw.lsn;
uint64_t reply_term = (uint64_t)war_raw.term;
int success = war_raw.success ? 1 : 0;
capn_free(&capn_ctx);
@ -4225,6 +4306,25 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx)
repl_on_ack(&c->persist.repl, ep, acked_lsn);
TRACE("repl_ack_decode_and_dispatch: WAL ACK from standby"
" lsn=%lu success=1", (unsigned long)acked_lsn);
/* ha-term-safety task 3.3 (R4.1/R4.5): route the ACK through the
* raft library so it updates match_index[]/next_index[] and
* recomputes the current-term-gated commit (raft_advance_commit).
* This fires on_commit_advanced (task 2.4) ONLY when a quorum
* agrees on a CURRENT-TERM entry — making the library the live
* commit source alongside the durable barrier; a stale-term reply
* is dropped inside the library. Driven on the same iomgr ACK
* path; neither library call performs I/O.
*
* Resolve the ABSOLUTE library peer index per endpoint type FIRST
* (remediation): this branch fires for BOTH promoted-vote eps
* (vctx->ctx != NULL, peer_idx is a relative votes index) AND
* heartbeat standby eps (vctx->ctx == NULL, peer_idx is already
* absolute) — see the authoritative scan at ha.c:6187-6189. A
* blind remap would double-map the heartbeat path onto the wrong
* (or out-of-range) match_index[] slot. */
int ack_abs_idx = ha_repl_ack_abs_peer_idx(
&c->ha, /*ep_is_vote*/ vctx->ctx != NULL, vctx->peer_idx);
ha_raft_on_repl_ack(c, ack_abs_idx, acked_lsn, reply_term);
/* ha-convergence-recovery task 4.2 (D6, Req 5.1/5.2/7.1):
* ACK-driven catch-up — continue streaming to a standby still
* behind target, or re-include it at parity. Shares the exact