myxce/.wolf/bugs/bug-1915.md

---
id: "bug-1915"
status: "fixed"
status_updated: "2026-10-04"
status_note: ""
component: "controller"
severity: "wrong-result"
timestamp: "2026-10-03T16:10:25Z"
last_seen: "2026-10-03T16:10:25Z"
occurrences: 1
files: ["src/controller/ha.c"]
tags: ["ha", "raft", "election", "prevote"]
related: ["bug-1913"]
fix_commits: ["88a8bf8a"]
gitea_issue: 338
gitea_synced_status: "fixed"
gitea_synced_hash: "28f363928651"
---

# bug-1915: two PreVote wins in one election replace elect_ctx; term-2 leader then finds no vote eps to promote

## Description
**Observed:**
In one primary log captured during the bug-1913 investigation (late-joiner election under 40-spinner CPU load), "PreVote round WON" appears twice (prospective terms 2 and 3). The second real election replaced `ha->elect_ctx`, and the term-2 `raft_cb_become_leader` then found no vote endpoints to promote into standby channels, so both standbys had to register later as late joiners (bug-1913). A term-2 leader was also logged after the term-3 election had started.

**Expected:** one election round per prospective term; a node that becomes leader promotes the vote connections from the election that elected it (or the stale election cannot complete leadership).
**Impact:** unconfirmed (single sample). It is the trigger for late-joiner elections (bug-1913, now mitigated by an immediate catch-up probe), and a leader logged for an older term after a newer election started needs a Raft-safety check (term monotonicity, which election's votes made it leader).
**Found by:** test (bug-1913 investigation, 2026-10-03)

## Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (xce.build)
- Code: scheduler-framework @ 6d9ef3d0-dirty
- Deployment: local 3-node HA (xceadmin live HA fixture), 40 busy loops on 32 cores

## Reproduction
1. Loop `src/xceadmin/tests/test_e2e_live_ha_push.py` under heavy CPU load with `XCE_KEEP_DEPLOY_DIR=1`.
2. In the elected primary's log, look for two "PreVote round WON" lines in one election and a `raft_cb_become_leader` that promotes no vote endpoints.
- Frequency: 1/40 runs, 2026-10-03 (single log sample under 40-spinner load; not yet reproduced deliberately)
- Evidence: not kept (scratchpad loop logs from the bug-1913 investigation)

## Root cause
A PreVote round could be decided more than once, and stale election traffic was not tied to the election it belonged to:
- `raft_on_pre_vote_reply` kept tallying after a round was already decided, so a late grant after WON re-ran the WON path and started a second real election (prospective term 3) that replaced `ha->elect_ctx` while the term-2 election was still completing.
- `raft_cb_become_leader` promoted vote endpoints from whatever `elect_ctx` was current, not the one that elected it, so the term-2 leader found the term-3 record (no granted eps) and promoted nobody (the bug-1913 late-joiner trigger).
- Related gaps found in review: a PreVote round could start while a real election was running or on a leader; replies, timeouts and endpoint errors from an older round (or a PreVote reply arriving on a real-vote endpoint, and vice versa) were counted against the current one; `raft_on_leader_lost` cleared `voted_for` within the same term, so a delayed same-term RequestVote from another candidate could be granted (two votes in one term); the failover stagger started PreVote inline on the delaymgr thread.

## Fix
- raft (src/lib/raft): a reply with no round in progress is a no-op, so a round is decided at most once. `raft_on_leader_lost` keeps `voted_for` and records `leader_lost_term`; `raft_check_and_start_election` starts the delayed election only while still a FOLLOWER in that term.
- controller (src/controller/ha.c/.h): every PreVote round has an id, and every PreVote endpoint carries its round and kind. Replies, round timeouts and endpoint errors from another round or of the other kind are dropped. Stale real-election replies (other elect_term) are not counted; a higher term still steps down. A PreVote round starts only on a follower with no election in progress, and each open round always has its timeout timer (arm failure closes the round). `raft_cb_become_leader` promotes only with THIS election's vote record (stamped carrier, own term, majority); otherwise it refuses, steps down to FOLLOWER in the same term and re-arms. A cancelled round timer returns without taking raft_lock. The failover stagger hands the PreVote start to a vtask (delaymgr stays trigger-only), re-checking that the primary is still gone, with an inline fallback when there is no pool.

## Verification
- Test: src/controller/testsuite/test_ha_prevote.c: `test_bug1915_late_grant_after_won_is_ignored`, `test_bug1915_late_deny_after_won_is_ignored`, plus the gapA/gapB/gapD/gapE, become_leader_*, r2_f* and final_f* cases; src/lib/testsuite/test_raft.c: `test_leader_lost_keeps_voted_for`, `test_leader_lost_same_term_vote_from_other_denied`, `test_check_and_start_after_leader_lost_same_term_starts`, `test_check_and_start_skips_when_not_follower`. Written ahead of each change in the 2026-10-03 fix sessions (RED/GREEN per the session log; RED not re-run on 2026-10-04).
- Suites: `xce-build.sh all` asan, 127/127 meson suites OK + Go tests ok (2026-10-04, together with bug-1919).
- Live: 2026-10-03: `/xce-test-ha` chaotic 6 and loaded 8 rounds PASS; the sequential 20-round run failed on bug-1919 (fence), with a clean single-decision election timeline. 2026-10-04: 4x sequential 20 rounds, 3 PASS, 1 FAIL on bug-1920 (catch-up stream stall after a log-mismatch NACK; under investigation, relation to this change not established). Across the four 2026-10-04 runs, 94 PreVote rounds were decided WON and none twice (one "PreVote round N WON" per pid+round+term).

## References
- Related: none