fix(ha-election,ha-rejoin): stop two infinite-election loops on failover and restart

Two distinct HA livelocks, both verified on a live 3-node ASAN deploy
(kill+restart of primary and standby, plus SIGSTOP wedge injection).

bug-339 — election freshness-floor asymmetry (RDHUP failover path):
After the primary dies the two survivors could never elect a leader — both
only ever became follower while terms climbed in lockstep. The RDHUP candidate
advertised c->persist.committed_lsn, which a standby never advances (only the
primary commit engine writes it), while every voter computed its freshness floor
from max(applied_lsn, last_known_committed_lsn). So each survivor advertised a
stale-low LSN against a high voter floor and the freshness gate denied on both
sides. Fix: compute the candidate LSN via ha_vote_self_freshness so the
advertised value equals the voter floor (ha_on_primary_ctrl_fd_rdhup); also route
the boot-path election (myapp_run) through the same helper for symmetry.

bug-340 — wedged standby endpoint suppressed reconnect (rejoin path):
A killed/restarted peer's outbound endpoint on the primary could wedge (epoll
re-arm ENOENT) yet stay in standby_endpoints, so primary_heartbeat_fn's reconnect
scan treated the peer as still-connected and never re-established a ctrlhello
channel — the restarted node never heard from the primary and re-elected every
heartbeat_timeout (term climbing unbounded). xce_nop_req returns success on
enqueue even when the send can't progress, so the heartbeat layer never noticed.
Fix: primary_heartbeat_fn now evicts a standby that was responsive once
(ts_last_in > 0) but has been silent past heartbeat_timeout — symmetric with the
standby's own liveness poll — and the reconnect scan re-establishes the channel.
Uses existing endpoint_t.ts_last_in (no libnetio change).

Tests (RED->GREEN):
- test_ha_rdhup_integration: candidate advertises freshness floor, not stale committed_lsn
- test_ha_liveness: evicts_silent_standby + keeps_live/keeps_unconfirmed (no false eviction)
This commit is contained in:
Rongsong Shen 2026-06-15 23:31:26 +00:00
parent ca1d9ae42d
commit a735a880ae
4 changed files with 271 additions and 10 deletions

View file

@ -213,9 +213,17 @@ int myapp_run(myconf_t *myconf)
}
}
uint64_t my_tag = myconf->config_version_tag;
uint64_t my_lsn = (uint64_t)atomic_load_explicit(&c.ha.applied_lsn,
memory_order_acquire);
/* Advertise the EFFECTIVE freshness floor — identical to the voter-side
* floor (cluster.c OPCode_requestVote) and to the RDHUP failover path
* (ha_on_primary_ctrl_fd_rdhup): max(applied_lsn, last_known_committed_lsn)
* at the effective config tag. Candidate-advertised LSN must equal what a
* peer would compute as its voter floor, otherwise two equally-fresh nodes
* mutually deny each other's votes and no leader is ever elected. */
uint64_t local_tag = myconf->config_version_tag;
uint64_t local_lsn = (uint64_t)atomic_load_explicit(&c.ha.applied_lsn,
memory_order_acquire);
uint64_t my_tag = 0, my_lsn = 0;
ha_vote_self_freshness(&c.ha, local_tag, local_lsn, &my_tag, &my_lsn);
/* Initialize raft state machine once with cluster topology.
* Must happen before any election or watchdog path that calls