lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha-config-bootstrap,ha-repl-hardening): config-less node must not win election

Browse source 
Root cause (found via ha-convergence-recovery task 8 live 3-node E2E): a
controller with no local config (config_version_tag==0) could become PRIMARY
and push an empty config, breaking replication.

Two upstream election-safety gaps, both outside ha-convergence-recovery:

1. RDHUP re-election bypassed the no-config 'voter only' guard. The startup
   path (ha_start_election_async) refuses to start an election when
   my_config_tag==0 && n_peers>1, but the RDHUP failover path
   (ha_on_primary_ctrl_fd_rdhup) called raft_on_leader_lost unconditionally.
   Fix: mirror the guard before elect_ctx alloc / hook / raft_on_leader_lost;
   a config-less node stays CTRL_STANDBY (voter only) and grants the
   configured peer instead of starting (and winning) its own election.

2. Voter freshness gate compared the candidate only against the voter's LOCAL
   config, ignoring the cluster's known committed state. Fix: ha_vote_self_freshness
   floors the comparison at max(local, learned-committed config_tag / committed_lsn),
   so a voter denies any candidate whose config/cluster state is outdated vs
   what the cluster has committed. Strictly additive (never denies the freshest
   node — Leader Completeness preserves liveness).

TDD: test_rdhup_noconfig_node_does_not_elect + configured-node contrast;
test_ha_vote_gate learned-floor deny + liveness + backward-compat. Full
controller suite green under ASAN. commit.c and ha-convergence-recovery
files untouched.
This commit is contained in:
Rongsong Shen 2026-06-14 08:58:08 +00:00
parent 513b965f98
commit c97c7632e9
6 changed files with 439 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

69
src/controller/ha.c
View file

@ -1982,6 +1982,30 @@ static int ha_heartbeat_is_fresh(ha_state_t *ha)
return ms < (long)ha->config.heartbeat_timeout_ms;
}
/* ── Voter freshness floor (election safety) ───────────────────────────────
*
* Compute the voter's effective freshness floor as the per-field max of its
* local state and its learned-committed cluster state. Outdated candidates
* (below this floor) are denied so the cluster can never be rolled back below
* what it has already committed. See ha.h for full contract.
*/
void ha_vote_self_freshness(ha_state_t *ha,
uint64_t local_config_tag,
uint64_t local_applied_lsn,
uint64_t *out_tag,
uint64_t *out_lsn)
{
uint64_t learned_tag =
atomic_load_explicit(&ha->committed_config_tag, memory_order_acquire);
uint64_t learned_lsn =
atomic_load_explicit(&ha->last_known_committed_lsn, memory_order_acquire);
if (out_tag)
*out_tag = (local_config_tag > learned_tag) ? local_config_tag : learned_tag;
if (out_lsn)
*out_lsn = (local_applied_lsn > learned_lsn) ? local_applied_lsn : learned_lsn;
}
/* ── Vote Handler (Req 2.2–2.6, 6.1, 6.2, 6.4, 9.1) ────────────────────────
*
* handle_request_vote: process an incoming OP_requestVote RPC.
@ -2073,15 +2097,20 @@ int handle_request_vote(ha_state_t *ha,
return 0;
}
/* Req 2.4: freshness gate
* Deny if candidate has a lower config_tag, or equal config_tag with lower LSN.
* Grant if candidate is strictly fresher or equally fresh.
*/
if (config_tag < my_config_tag) {
/* Req 2.4 + election-safety fix #2: freshness gate against the EFFECTIVE
* floor — the max of the voter's local state and its learned-committed
* cluster state. A candidate that is below what the cluster has already
* committed (config_tag or, at equal tag, LSN) would roll the cluster back
* and must be denied even when this voter's local state is colder.
* Deny if candidate has a lower config_tag, or equal config_tag with lower
* LSN; grant if strictly fresher or equally fresh. */
uint64_t eff_tag = 0, eff_lsn = 0;
ha_vote_self_freshness(ha, my_config_tag, my_last_lsn, &eff_tag, &eff_lsn);
if (config_tag < eff_tag) {
pthread_mutex_unlock(&ha->lock);
return 0;
}
if (config_tag == my_config_tag && last_lsn < my_last_lsn) {
if (config_tag == eff_tag && last_lsn < eff_lsn) {
pthread_mutex_unlock(&ha->lock);
return 0;
}
@ -3423,6 +3452,34 @@ void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c)
if (committed > 0)
persist_truncate_wal_above(&c->persist, committed);
/* Election-SAFETY guard (ha-config-bootstrap Req 2.8 / ha-repl-hardening
* Req 10.5): a config-less node (config_version_tag == 0) must NEVER start
* an RDHUP re-election while a configured peer exists. Two such nodes would
* grant each other (the freshness gate only denies a LOWER tag; tag0==tag0
* with equal lsn is granted), forming a majority and electing a config-less
* primary over the node that actually holds the config — which then pushes
* an empty config and breaks replication.
*
* Mirror the proven ha_start_election_async voter-only guard: skip the
* elect_ctx alloc + the election trigger (raft_on_leader_lost) entirely and
* stay a passive voter (CTRL_STANDBY). handle_request_vote still GRANTS
* incoming vote requests to the fresher configured peer regardless of role,
* and this node transitions/fetches config when the configured winner
* connects and pushes config via ctrlhello.
*
* Single-node (candidates_count == 1) is unaffected: n_peers > 1 is false,
* so a lone node still self-elects regardless of tag. */
uint64_t my_config_tag = (c->myconf) ? c->myconf->config_version_tag : 0;
int n_peers_guard = (c->myconf) ? c->myconf->peers_cfg.candidates_count : 1;
if (my_config_tag == 0 && n_peers_guard > 1) {
INFO("ha_on_primary_ctrl_fd_rdhup: no local config (tag=0, n_peers=%d) — "
"voter only, not starting RDHUP re-election; awaiting configured "
"primary (ha-config-bootstrap Req 2.8)", n_peers_guard);
atomic_store_explicit(&ha->role, (int32_t)CTRL_STANDBY,
memory_order_release);
return;
}
/* Task 6.3 / 6.7 fix: delegate RDHUP election trigger to raft library.
*
* raft_on_leader_lost: