---
id: "bug-606"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "hang"
timestamp: "2026-06-16T17:46:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c (ha_begin_failover", "ha_election_stagger_ms", "ha_failover_stagger_fn); src/controller/ha.h; test: src/controller/testsuite/test_ha_role.c"]
tags: ["ha", "raft", "election", "split-vote", "dueling-candidates", "livelock", "randomized-timeout", "failover", "stagger", "jitter", "FIXED"]
related: ["bug-605"]
legacy: true
---

# bug-606: After killing the primary (ctrl-1), the two surviving standbys (ctrl-2,ctrl-3) livelock in…

## Description
After killing the primary (ctrl-1), the two surviving standbys (ctrl-2,ctrl-3) livelock in synchronized elections: PreVote win → CTRL_ELECTING → become_follower (term bumped) → ~10s → repeat, term climbing unbounded (3→9→∞), no new primary ever elected. Reproduce: deploy 3-node HA, elect primary, submit 3-5 jobs, kill ctrl-1.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
Dueling-candidates split-vote livelock — Raft §5.2 randomized election timeout is missing on the FAILOVER path. Both survivors lose the primary at the same instant; their liveness polls fire on the identical fixed heartbeat_timeout, so they start PreVote+real-election in perfect lock-step (ms-aligned timestamps). Both win PreVote (neither sees a primary, so each grants the other — PreVote cannot break this tie), then both campaign at the same term, each votes for itself and denies the other (quorum 2, only 2 live nodes) → split vote → election times out (fixed election_timeout_ms, raft.c:105, NO jitter) → become_follower, term+1 → repeat forever. The priority/jitter config (election_jitter_max_ms, election_priority_step_ms) is applied ONLY at startup (ha_wait_for_heartbeat_or_timeout); the failover re-election path (liveness_poll_fn → ha_on_primary_ctrl_fd_rdhup → PreVote → ha_start_election_async → raft_start_election) has no de-synchronization. raft_on_leader_lost DOES have a priority stagger (raft.c:421) but the failover loop does not go through it.

## Fix
Added ha_election_stagger_ms() (ha.c/ha.h): per-node failover delay = my_priority*election_priority_step_ms + random[0,election_jitter_max_ms], re-randomized each call (mirrors the startup stagger). ha_begin_failover() now schedules the PreVote start via delaymgr after that stagger (production only; c==NULL/no-delaymgr stays synchronous so existing tests are unaffected). The delaymgr callback ha_failover_stagger_fn guards with role==STANDBY && ha_heartbeat_is_fresh → skips PreVote if a primary connected during the stagger. Effect: two survivors no longer campaign in lock-step; the higher-priority survivor wins and its heartbeats reset the other. TDD: test_election_stagger_desyncs_by_priority (bounds + step>jitter ordering). VERIFIED on kill-primary repro: ctrl-2 (stagger 630ms) won → PRIMARY term=1; ctrl-3 (stagger 1072ms) received ctrl-2's ctrlhello first → STANDBY, then its stagger timer hit the guard and skipped PreVote ('a primary is alive — skipping PreVote'). No term inflation, no livelock. NOTE: optional raft.c:105 give-up-watchdog randomization NOT needed — the failover-start stagger alone resolves it; left as future defense-in-depth.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-605