---
id: "bug-1423"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "hang"
timestamp: "2026-07-09T13:58:16.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/ha.c"]
tags: ["ha", "raft", "election", "liveness-deadlock", "vote-request", "dead-peer", "quorum", "permanent-stall"]
related: ["bug-1417", "bug-1418", "bug-1421"]
legacy: true
---

# bug-1423: 30-round sequential HA soak FAIL at round 5: post-failover submit did not get 3 acks after 5 retry…

## Description
30-round sequential HA soak FAIL at round 5: post-failover submit did not get 3 acks after 5 retry attempts (15s+). Root-caused as a genuine permanent election deadlock, not a client-side false positive: ctrl-3 won PreVote and entered CTRL_ELECTING (real election, term=10) with ctrl-1 dead and ctrl-2 alive, then never showed another HA state-machine event for 2+ minutes (until the harness killed the cluster on FAIL) — stuck in CTRL_ELECTING forever. Meanwhile ctrl-2 endlessly retried its own PreVote every ~15-17s, correctly denied every time by ctrl-3 on freshness grounds, but that denial never unstuck ctrl-3's already-in-flight real election.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
raft_cb_send_vote_request (ha.c, sends the real RequestVote RPC to each peer) has two failure branches — endpoint_connect() failing and iomgr_monitor() failing — that are exactly what happens when the RPC target is a dead/unreachable peer (here, ctrl-1, killed at the start of round 5). The endpoint_connect failure branch only called the legacy handle_request_vote_reply() when `c == NULL` (unit-test mode) — in PRODUCTION (c always non-NULL) a connect failure to a dead peer was silently dropped with NO notification to the raft library at all. The iomgr_monitor failure branch called handle_request_vote_reply() unconditionally, including in production — but per vote_reply_dispatch's own design comment, handle_request_vote_reply is the LEGACY/ctx-done-signaling path; 'raft_on_vote_reply remains the SOLE production promotion driver' (vote_reply_dispatch calls ha_record_vote_reply + raft_on_vote_reply, NOT handle_request_vote_reply). Net effect: a candidate's vote request to a dead peer never registers as a 'reply' with the production raft library's raft_state_t.replies_received counter. raft.c's Step 6 fast-path ('all peers have replied, no quorum → become follower and retry', gated on replies_received == n_peers-1) can therefore NEVER trigger whenever any configured peer is down — with quorum=2 in a 3-node cluster and one peer dead, replies_received caps at 1 (only the live peer's real reply, if any) and can never reach n_peers-1=2. Recovery then depends ENTIRELY on the raw per-campaign election_timeout (5-10s, ha_jittered_election_delay) firing via delaymgr — in the observed failure, no election_timeout_fn trace appeared in the log at all for the full 2+ minute window, leaving the candidate permanently wedged.

## Fix
Routed both raft_cb_send_vote_request failure branches (connect failure and monitor failure) through vote_reply_dispatch(ha, ectx, votes_idx, term, vote_granted=0, 0, 0) in production (c != NULL) — the exact same call a genuine vote reply makes (ha_record_vote_reply + raft_on_vote_reply), so a dead/unreachable peer now correctly counts as 'replied, not granted' immediately on connect/monitor failure instead of only via the timeout. Test mode (c == NULL) keeps the legacy handle_request_vote_reply call unit tests depend on. Added a forward declaration of the previously-file-local-only vote_reply_dispatch near the top of ha.c so raft_cb_send_vote_request (defined much earlier in the file) can call it. Verified: full controller_unit_test suite clean (755 PASS / 0 Fail, exit 0) both before and after removing temporary diagnostic logging used during root-cause; re-ran the exact failing scenario — 30-round sequential HA soak PASS (93 jobs, zero loss, all 3 nodes led) and 30-round chaotic HA soak PASS (93 jobs, zero committed divergence, zero committed job loss) — both previously hit this stall reliably (round 5 of the sequential run, 100% reproduction before the fix).

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1417, bug-1418, bug-1421