--- id: "bug-1417" status: "fixed" status_updated: "2026-09-24" status_note: "" component: "controller" severity: "unknown" timestamp: "2026-07-07T09:30:00.000Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 1 files: ["src/controller/ha.c"] tags: ["vote", "raft", "self_idx", "index-mapping", "chaotic-failover", "off-by-one", "ha", "ha_repl_ack_abs_peer_idx", "election"] related: ["bug-828"] legacy: true --- # bug-1417: [xce-test-ha --chaotic] FAIL round 4/10: election stall after killing ctrl-1 — ctrl-2 PreVote wins… ## Description [xce-test-ha --chaotic] FAIL round 4/10: election stall after killing ctrl-1 — ctrl-2 PreVote wins but real election times out despite ctrl-3 granting RequestVote ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction _Not recorded (migrated from buglog.json)._ ## Root cause vote_reply_dispatch passes local votes_idx (compressed, excludes self) directly to raft_on_vote_reply which expects ABSOLUTE peer index (includes self). For self_idx=1 (ctrl-2), ctrl-3's vote (absolute=2, local=1) lands in vote_replies[1]=ctrl-2's own self-slot → treated as duplicate → vote silently dropped. With ctrl-1 dead in chaotic mode, ctrl-2 only has self-vote (1 < quorum 2) → eternal timeout. Self_idx=0 and self_idx=2 are partially/fully immune due to index alignment luck. ha_repl_ack_abs_peer_idx() already exists (ha.c:3779) with correct mapping and passing tests — vote_reply_dispatch just doesn't call it. ## Fix FIXED (src/controller/ha.c vote_reply_dispatch, ~line 5233): raft_on_vote_reply now receives ha_repl_ack_abs_peer_idx(ha, 1, idx) instead of the raw compressed local idx. ha_repl_ack_abs_peer_idx (ha.c:3779, already existed + unit-tested for the heartbeat-ACK path) performs the exact inverse of raft_cb_send_vote_request send-side remap: votes_idx < self_idx ? votes_idx : votes_idx + 1. ha_record_vote_reply (immediately preceding line) is UNCHANGED -- it correctly wants the compressed idx for ctx->votes[]. Confirmed pre_vote_reply_dispatch needs NO equivalent fix: raft_cb_send_pre_vote_request (ha.c:301) already stores the ABSOLUTE peer_idx in vctx->peer_idx for the PreVote path (explicit code comment there), matching raft_on_pre_vote_reply/pre_vote_replies[] absolute-index expectation -- only the real-vote send path (raft_cb_send_vote_request, ha.c:175) stores compressed. RED->GREEN: new unit test test_bug1417_middle_index_real_vote_from_peer2_not_dropped (test_ha_vote_send.c) drives a REAL 3-node election (self_idx=1) through raft_start_election then a genuine peer-2 grant via vote_reply_msg_handler_for_test (the actual decode/dispatch path, not the xce_vote_reply_hook bypass every other real-vote test in this file uses) -- reproduced Expected 2 Was 1 pre-fix, GREEN post-fix (votes_granted=2, role=RAFT_LEADER). Verified: full 79-group controller_unit_test sweep 0 new failures; 10-round --chaotic xce-test-ha soak PASS (committed_divergence=0, committed_job_loss=0, 18 jobs). Soak surfaced an UNRELATED, pre-existing --chaotic harness bug (run-ha-soak.sh run_chaotic_rounds: DOWN variable is not cleared on a skip-this-round no-primary-to-kill path, so a subsequent round calls start_node on the same already-running node without killing it first -- 6 stacked ctrl-2 processes observed piling up over rounds 5-10) -- confirmed via git diff this script was untouched by this fix, logged separately, NOT fixed here (out of bug-1417 boundary). ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-828