fix(ha-election,ha-rejoin): stop two infinite-election loops on failover and restart
Two distinct HA livelocks, both verified on a live 3-node ASAN deploy (kill+restart of primary and standby, plus SIGSTOP wedge injection). bug-339 — election freshness-floor asymmetry (RDHUP failover path): After the primary dies the two survivors could never elect a leader — both only ever became follower while terms climbed in lockstep. The RDHUP candidate advertised c->persist.committed_lsn, which a standby never advances (only the primary commit engine writes it), while every voter computed its freshness floor from max(applied_lsn, last_known_committed_lsn). So each survivor advertised a stale-low LSN against a high voter floor and the freshness gate denied on both sides. Fix: compute the candidate LSN via ha_vote_self_freshness so the advertised value equals the voter floor (ha_on_primary_ctrl_fd_rdhup); also route the boot-path election (myapp_run) through the same helper for symmetry. bug-340 — wedged standby endpoint suppressed reconnect (rejoin path): A killed/restarted peer's outbound endpoint on the primary could wedge (epoll re-arm ENOENT) yet stay in standby_endpoints, so primary_heartbeat_fn's reconnect scan treated the peer as still-connected and never re-established a ctrlhello channel — the restarted node never heard from the primary and re-elected every heartbeat_timeout (term climbing unbounded). xce_nop_req returns success on enqueue even when the send can't progress, so the heartbeat layer never noticed. Fix: primary_heartbeat_fn now evicts a standby that was responsive once (ts_last_in > 0) but has been silent past heartbeat_timeout — symmetric with the standby's own liveness poll — and the reconnect scan re-establishes the channel. Uses existing endpoint_t.ts_last_in (no libnetio change). Tests (RED->GREEN): - test_ha_rdhup_integration: candidate advertises freshness floor, not stale committed_lsn - test_ha_liveness: evicts_silent_standby + keeps_live/keeps_unconfirmed (no false eviction)
This commit is contained in:
parent
ca1d9ae42d
commit
a735a880ae
4 changed files with 271 additions and 10 deletions
|
|
@ -213,9 +213,17 @@ int myapp_run(myconf_t *myconf)
|
|||
}
|
||||
}
|
||||
|
||||
uint64_t my_tag = myconf->config_version_tag;
|
||||
uint64_t my_lsn = (uint64_t)atomic_load_explicit(&c.ha.applied_lsn,
|
||||
memory_order_acquire);
|
||||
/* Advertise the EFFECTIVE freshness floor — identical to the voter-side
|
||||
* floor (cluster.c OPCode_requestVote) and to the RDHUP failover path
|
||||
* (ha_on_primary_ctrl_fd_rdhup): max(applied_lsn, last_known_committed_lsn)
|
||||
* at the effective config tag. Candidate-advertised LSN must equal what a
|
||||
* peer would compute as its voter floor, otherwise two equally-fresh nodes
|
||||
* mutually deny each other's votes and no leader is ever elected. */
|
||||
uint64_t local_tag = myconf->config_version_tag;
|
||||
uint64_t local_lsn = (uint64_t)atomic_load_explicit(&c.ha.applied_lsn,
|
||||
memory_order_acquire);
|
||||
uint64_t my_tag = 0, my_lsn = 0;
|
||||
ha_vote_self_freshness(&c.ha, local_tag, local_lsn, &my_tag, &my_lsn);
|
||||
|
||||
/* Initialize raft state machine once with cluster topology.
|
||||
* Must happen before any election or watchdog path that calls
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue