--- id: "bug-660" status: "fixed" status_updated: "2026-09-24" status_note: "" component: "controller" severity: "unknown" timestamp: "2026-06-17T22:00:00Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 1 files: ["src/controller/ha.c"] tags: ["controller", "ha", "election", "prevote", "convergence", "fresh_heartbeat", "raft_cb_vote_eligible", "primary_ctrl_fd", "slow-election"] related: [] legacy: true --- # bug-660: Failover election takes minutes (11+ PreVote rounds) to converge ## Description Failover election takes minutes (11+ PreVote rounds) to converge. A standby keeps denying a legitimately-higher-term candidate's PreVote with reason=fresh_heartbeat for ~2.5 min AFTER its primary died (ctrl-3 killed 21:44:14, ctrl-2 logs 'PreVote DENIED ... reason=fresh_heartbeat (my_term=3)' until 21:46:58). ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction _Not recorded (migrated from buglog.json)._ ## Root cause The PreVote/vote eligibility gate raft_cb_vote_eligible (ha.c:311) denied on `role==CTRL_STANDBY && ha_heartbeat_is_fresh(ha)` — checking freshness ALONE. ha_heartbeat_is_fresh derives from ha->last_activity, but ha_arm_heartbeat_timeout_vtask resets last_activity=now on EVERY watchdog (re-)arm, and ha_prevote_rearm_liveness re-arms it after every lost PreVote. So a node with NO live primary perpetually refreshes its own 'fresh heartbeat' each failover cycle and denies all other candidates (reason=fresh_heartbeat), blocking convergence until it happens to restart. The sibling gate in handle_request_vote (ha.c:2205) already required `primary_ctrl_fd >= 0 && is_fresh`; raft_cb_vote_eligible was missing the primary_ctrl_fd guard. The existing fresh-deny tests even documented the intent ('the deny gate requires a live connection AND a fresh heartbeat', set primary_ctrl_fd=999), but production didn't enforce it. ## Fix Added `ha->primary_ctrl_fd >= 0 &&` to the ha.c:311 eligibility gate (mirrors ha.c:2205). primary_ctrl_fd is -1 on init / after the primary RDHUPs / after the liveness watchdog fires, and a real fd only via config_repl.c:1451 (first ctrlhello from the live primary) — so a dead/absent primary no longer masquerades as fresh and the standby GRANTS, converging in ~1 round. Kept cluster.c:526 denial-reason logging in sync. TDD: test_pre_vote_dead_primary_grants_despite_fresh_activity (fresh last_activity + primary_ctrl_fd=-1 -> GRANT); verified RED (Expected 1 Was 0) without the guard, GREEN with it, and the 3 existing fresh-deny tests (fd=999) unaffected. Full controller suite + libxce 169/0, zero regressions. ## Verification _Not recorded (migrated from buglog.json)._ ## References _Not recorded (migrated from buglog.json)._