lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha-election,ha-rejoin): stop two infinite-election loops on failover and restart

Browse source 
Two distinct HA livelocks, both verified on a live 3-node ASAN deploy
(kill+restart of primary and standby, plus SIGSTOP wedge injection).

bug-339 — election freshness-floor asymmetry (RDHUP failover path):
After the primary dies the two survivors could never elect a leader — both
only ever became follower while terms climbed in lockstep. The RDHUP candidate
advertised c->persist.committed_lsn, which a standby never advances (only the
primary commit engine writes it), while every voter computed its freshness floor
from max(applied_lsn, last_known_committed_lsn). So each survivor advertised a
stale-low LSN against a high voter floor and the freshness gate denied on both
sides. Fix: compute the candidate LSN via ha_vote_self_freshness so the
advertised value equals the voter floor (ha_on_primary_ctrl_fd_rdhup); also route
the boot-path election (myapp_run) through the same helper for symmetry.

bug-340 — wedged standby endpoint suppressed reconnect (rejoin path):
A killed/restarted peer's outbound endpoint on the primary could wedge (epoll
re-arm ENOENT) yet stay in standby_endpoints, so primary_heartbeat_fn's reconnect
scan treated the peer as still-connected and never re-established a ctrlhello
channel — the restarted node never heard from the primary and re-elected every
heartbeat_timeout (term climbing unbounded). xce_nop_req returns success on
enqueue even when the send can't progress, so the heartbeat layer never noticed.
Fix: primary_heartbeat_fn now evicts a standby that was responsive once
(ts_last_in > 0) but has been silent past heartbeat_timeout — symmetric with the
standby's own liveness poll — and the reconnect scan re-establishes the channel.
Uses existing endpoint_t.ts_last_in (no libnetio change).

Tests (RED->GREEN):
- test_ha_rdhup_integration: candidate advertises freshness floor, not stale committed_lsn
- test_ha_liveness: evicts_silent_standby + keeps_live/keeps_unconfirmed (no false eviction)
This commit is contained in:
Rongsong Shen 2026-06-15 23:31:26 +00:00
parent ca1d9ae42d
commit a735a880ae
4 changed files with 271 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

73
src/controller/ha.c
View file

@ -725,7 +725,8 @@ static void raft_cb_become_follower(void *ctx, uint64_t term, int leader_idx)
}
/* Arm heartbeat watchdog — fires ha_on_primary_ctrl_fd_rdhup if primary
* sends no ctrlhello within election_timeout_ms. */
* sends no ctrlhello within election_timeout_ms. (This also resets
* last_activity = now, granting a full grace window before the first poll.) */
ha_arm_heartbeat_timeout_vtask(ha, c);
}
@ -3557,6 +3558,28 @@ void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c)
* so that the subsequent raft_start_election finds a valid context. */
ha->raft_c = c;
/* Candidate freshness for the RequestVote payload. This MUST mirror the
* voter-side freshness floor (cluster.c OPCode_requestVote): the effective
* floor is max(applied_lsn, last_known_committed_lsn) at the EFFECTIVE
* config tag. A standby never advances persist.committed_lsn (only the
* primary's commit engine writes it) — it advances applied_lsn (WAL
* catch-up) and last_known_committed_lsn (heartbeat replies). Advertising
* persist.committed_lsn here makes every surviving standby advertise a
* stale-low LSN while each peer computes a HIGH floor from applied_lsn;
* the freshness gate (cand_lsn < my_lsn) then trips on both sides, each
* denies the other's vote, no candidate reaches quorum, and the cluster
* spins in an infinite RDHUP→ELECTING→STANDBY loop after the primary dies.
* Using ha_vote_self_freshness makes advertised == voter-floor, so the
* freshest node is always granted by every peer. (Matches the boot-path
* election in myapp_run, which advertises applied_lsn.) */
uint64_t cand_tag = 0, cand_lsn = 0;
if (c->myconf) {
uint64_t local_tag = c->myconf->config_version_tag;
uint64_t local_lsn = atomic_load_explicit(&ha->applied_lsn,
memory_order_acquire);
ha_vote_self_freshness(ha, local_tag, local_lsn, &cand_tag, &cand_lsn);
}
/* Allocate fresh elect_ctx so raft_cb_send_vote_request can read my_tag/my_lsn */
if (c && c->myconf) {
int n_peers = c->myconf->peers_cfg.candidates_count;
@ -3564,9 +3587,8 @@ void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c)
ha_elect_ctx_t *new_ctx = calloc(1, sizeof(*new_ctx) +
(size_t)n_other * sizeof(new_ctx->votes[0]));
if (new_ctx) {
uint64_t my_tag = c->myconf->config_version_tag;
uint64_t my_lsn = atomic_load_explicit(&c->persist.committed_lsn,
memory_order_acquire);
uint64_t my_tag = cand_tag;
uint64_t my_lsn = cand_lsn;
new_ctx->n_peers = n_other;
new_ctx->votes_granted = 1; /* self-vote placeholder */
new_ctx->result = -1;
@ -3607,9 +3629,8 @@ void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c)
c->myhost ? c->myhost->name : "") != 0)
peer_ptrs[np++] = &cfg->candidates[i];
}
uint64_t my_tag = c->myconf->config_version_tag;
uint64_t my_lsn = atomic_load_explicit(&c->persist.committed_lsn,
memory_order_acquire);
uint64_t my_tag = cand_tag;
uint64_t my_lsn = cand_lsn;
const char *my_name = c->myconf ? c->myconf->name : "";
ha_rdhup_election_hook(ha, c, np, peer_ptrs, my_tag, my_lsn, my_name);
}
@ -4094,6 +4115,17 @@ static int primary_heartbeat_fn(void *arg)
interval_s = HA_DEFAULT_HEARTBEAT_INTERVAL_MS / 1000;
if (ha->standby_endpoints) {
/* Liveness eviction threshold (wall clock, matches ep->ts_last_in). */
time_t now_wall = time(NULL);
uint32_t timeout_s = ha->config.heartbeat_timeout_ms / 1000;
if (timeout_s == 0)
timeout_s = HA_DEFAULT_HEARTBEAT_TIMEOUT_MS / 1000;
/* Endpoints evicted this tick; their repl deregistration runs AFTER
* ha->lock is released (repl lock-ordering — mirrors the err handlers). */
endpoint_t *evicted[MAX_HA_CANDIDATES];
int n_evicted = 0;
pthread_mutex_lock(&ha->lock);
int n = ha->standby_endpoints->count;
for (int i = 0; i < n; i++) {
@ -4101,6 +4133,30 @@ static int primary_heartbeat_fn(void *arg)
if (!ep) continue;
standby_ep_ctx_t *sctx = (standby_ep_ctx_t *)ep->data;
if (!sctx) continue;
/* Liveness eviction (symmetric with the standby's liveness poll):
* a standby that was responsive at least once (ts_last_in > 0) but
* has sent nothing for longer than heartbeat_timeout is dead or was
* restarted. A wedged/dead endpoint left in the array makes the
* reconnect scan below treat the peer as still-connected, so the
* restarted node never receives a fresh ctrlhello and loops
* elections forever (terms climbing unbounded). Evict it here; the
* reconnect scan re-establishes a fresh channel on this/next tick.
* Removal from standby_endpoints is the load-bearing action — the
* endpoint object itself is reclaimed by its err handler when the
* dead connection finally surfaces an epoll event (bounded). */
if (ep->ts_last_in > 0 &&
(now_wall - ep->ts_last_in) > (time_t)timeout_s) {
WARN("HA: standby[%d] silent for %lds (timeout=%us) — evicting "
"dead endpoint, will reconnect",
i, (long)(now_wall - ep->ts_last_in), timeout_s);
zarray_del(ha->standby_endpoints, i);
if (n_evicted < MAX_HA_CANDIDATES)
evicted[n_evicted++] = ep;
i--; n--;
continue;
}
time_t elapsed = now - sctx->last_push_time;
if (elapsed >= (time_t)interval_s) {
#ifdef XCE_TESTING
@ -4115,6 +4171,9 @@ static int primary_heartbeat_fn(void *arg)
}
}
pthread_mutex_unlock(&ha->lock);
for (int k = 0; k < n_evicted; k++)
repl_remove_standby(&c->persist.repl, evicted[k]);
}
}