| id |
status |
status_updated |
status_note |
component |
severity |
timestamp |
last_seen |
occurrences |
files |
tags |
related |
fix_commits |
gitea_issue |
gitea_synced_status |
gitea_synced_hash |
| bug-1915 |
fixed |
2026-10-04 |
|
controller |
wrong-result |
2026-10-03T16:10:25Z |
2026-10-03T16:10:25Z |
1 |
|
|
|
|
338 |
fixed |
28f363928651 |
bug-1915: two PreVote wins in one election replace elect_ctx; term-2 leader then finds no vote eps to promote
Description
Observed:
In one primary log captured during the bug-1913 investigation (late-joiner election under 40-spinner CPU load), "PreVote round WON" appears twice (prospective terms 2 and 3). The second real election replaced ha->elect_ctx, and the term-2 raft_cb_become_leader then found no vote endpoints to promote into standby channels, so both standbys had to register later as late joiners (bug-1913). A term-2 leader was also logged after the term-3 election had started.
Expected: one election round per prospective term; a node that becomes leader promotes the vote connections from the election that elected it (or the stale election cannot complete leadership).
Impact: unconfirmed (single sample). It is the trigger for late-joiner elections (bug-1913, now mitigated by an immediate catch-up probe), and a leader logged for an older term after a newer election started needs a Raft-safety check (term monotonicity, which election's votes made it leader).
Found by: test (bug-1913 investigation, 2026-10-03)
Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (xce.build)
- Code: scheduler-framework @ 6d9ef3d0-dirty
- Deployment: local 3-node HA (xceadmin live HA fixture), 40 busy loops on 32 cores
Reproduction
- Loop
src/xceadmin/tests/test_e2e_live_ha_push.py under heavy CPU load with XCE_KEEP_DEPLOY_DIR=1.
- In the elected primary's log, look for two "PreVote round WON" lines in one election and a
raft_cb_become_leader that promotes no vote endpoints.
- Frequency: 1/40 runs, 2026-10-03 (single log sample under 40-spinner load; not yet reproduced deliberately)
- Evidence: not kept (scratchpad loop logs from the bug-1913 investigation)
Root cause
A PreVote round could be decided more than once, and stale election traffic was not tied to the election it belonged to:
raft_on_pre_vote_reply kept tallying after a round was already decided, so a late grant after WON re-ran the WON path and started a second real election (prospective term 3) that replaced ha->elect_ctx while the term-2 election was still completing.raft_cb_become_leader promoted vote endpoints from whatever elect_ctx was current, not the one that elected it, so the term-2 leader found the term-3 record (no granted eps) and promoted nobody (the bug-1913 late-joiner trigger).- Related gaps found in review: a PreVote round could start while a real election was running or on a leader; replies, timeouts and endpoint errors from an older round (or a PreVote reply arriving on a real-vote endpoint, and vice versa) were counted against the current one;
raft_on_leader_lost cleared voted_for within the same term, so a delayed same-term RequestVote from another candidate could be granted (two votes in one term); the failover stagger started PreVote inline on the delaymgr thread.
Fix
- raft (src/lib/raft): a reply with no round in progress is a no-op, so a round is decided at most once.
raft_on_leader_lost keeps voted_for and records leader_lost_term; raft_check_and_start_election starts the delayed election only while still a FOLLOWER in that term.
- controller (src/controller/ha.c/.h): every PreVote round has an id, and every PreVote endpoint carries its round and kind. Replies, round timeouts and endpoint errors from another round or of the other kind are dropped. Stale real-election replies (other elect_term) are not counted; a higher term still steps down. A PreVote round starts only on a follower with no election in progress, and each open round always has its timeout timer (arm failure closes the round).
raft_cb_become_leader promotes only with THIS election's vote record (stamped carrier, own term, majority); otherwise it refuses, steps down to FOLLOWER in the same term and re-arms. A cancelled round timer returns without taking raft_lock. The failover stagger hands the PreVote start to a vtask (delaymgr stays trigger-only), re-checking that the primary is still gone, with an inline fallback when there is no pool.
Verification
- Test: src/controller/testsuite/test_ha_prevote.c:
test_bug1915_late_grant_after_won_is_ignored, test_bug1915_late_deny_after_won_is_ignored, plus the gapA/gapB/gapD/gapE, become_leader_, r2_f and final_f* cases; src/lib/testsuite/test_raft.c: test_leader_lost_keeps_voted_for, test_leader_lost_same_term_vote_from_other_denied, test_check_and_start_after_leader_lost_same_term_starts, test_check_and_start_skips_when_not_follower. Written ahead of each change in the 2026-10-03 fix sessions (RED/GREEN per the session log; RED not re-run on 2026-10-04).
- Suites:
xce-build.sh all asan, 127/127 meson suites OK + Go tests ok (2026-10-04, together with bug-1919).
- Live: 2026-10-03:
/xce-test-ha chaotic 6 and loaded 8 rounds PASS; the sequential 20-round run failed on bug-1919 (fence), with a clean single-decision election timeline. 2026-10-04: 4x sequential 20 rounds, 3 PASS, 1 FAIL on bug-1920 (catch-up stream stall after a log-mismatch NACK; under investigation, relation to this change not established). Across the four 2026-10-04 runs, 94 PreVote rounds were decided WON and none twice (one "PreVote round N WON" per pid+round+term).
References
