lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(buglog): bug-1915 and bug-1919 fixed; log bug-1920

Browse source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-04 09:14:58 +00:00
parent 0ab12074f5
commit f01bb7de34
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
4 changed files with 205 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

21
.wolf/bugs/bug-1915.md
View file

@ -1,8 +1,8 @@
---
id: "bug-1915"
status: "investigating"
status_updated: "2026-10-03"
status_note: "single log sample; reproduce under load and check election/term handling around elect_ctx replacement"
status: "fixed"
status_updated: "2026-10-04"
status_note: ""
component: "controller"
severity: "wrong-result"
timestamp: "2026-10-03T16:10:25Z"
@ -11,6 +11,7 @@ occurrences: 1
files: ["src/controller/ha.c"]
tags: ["ha", "raft", "election", "prevote"]
related: ["bug-1913"]
fix_commits: ["88a8bf8a"]
---
# bug-1915: two PreVote wins in one election replace elect_ctx; term-2 leader then finds no vote eps to promote
@ -38,15 +39,19 @@ In one primary log captured during the bug-1913 investigation (late-joiner elect
- Evidence: not kept (scratchpad loop logs from the bug-1913 investigation)
## Root cause
TODO: required for fixed/partial
A PreVote round could be decided more than once, and stale election traffic was not tied to the election it belonged to:
- `raft_on_pre_vote_reply` kept tallying after a round was already decided, so a late grant after WON re-ran the WON path and started a second real election (prospective term 3) that replaced `ha->elect_ctx` while the term-2 election was still completing.
- `raft_cb_become_leader` promoted vote endpoints from whatever `elect_ctx` was current, not the one that elected it, so the term-2 leader found the term-3 record (no granted eps) and promoted nobody (the bug-1913 late-joiner trigger).
- Related gaps found in review: a PreVote round could start while a real election was running or on a leader; replies, timeouts and endpoint errors from an older round (or a PreVote reply arriving on a real-vote endpoint, and vice versa) were counted against the current one; `raft_on_leader_lost` cleared `voted_for` within the same term, so a delayed same-term RequestVote from another candidate could be granted (two votes in one term); the failover stagger started PreVote inline on the delaymgr thread.
## Fix
TODO: required for fixed/partial
- raft (src/lib/raft): a reply with no round in progress is a no-op, so a round is decided at most once. `raft_on_leader_lost` keeps `voted_for` and records `leader_lost_term`; `raft_check_and_start_election` starts the delayed election only while still a FOLLOWER in that term.
- controller (src/controller/ha.c/.h): every PreVote round has an id, and every PreVote endpoint carries its round and kind. Replies, round timeouts and endpoint errors from another round or of the other kind are dropped. Stale real-election replies (other elect_term) are not counted; a higher term still steps down. A PreVote round starts only on a follower with no election in progress, and each open round always has its timeout timer (arm failure closes the round). `raft_cb_become_leader` promotes only with THIS election's vote record (stamped carrier, own term, majority); otherwise it refuses, steps down to FOLLOWER in the same term and re-arms. A cancelled round timer returns without taking raft_lock. The failover stagger hands the PreVote start to a vtask (delaymgr stays trigger-only), re-checking that the primary is still gone, with an inline fallback when there is no pool.
## Verification
- Test: TODO: name + file; RED before the fix, GREEN after
- Suites: TODO: suite results and build profile
- Live: TODO: soak/deploy run, or n/a and why
- Test: src/controller/testsuite/test_ha_prevote.c: `test_bug1915_late_grant_after_won_is_ignored`, `test_bug1915_late_deny_after_won_is_ignored`, plus the gapA/gapB/gapD/gapE, become_leader_*, r2_f* and final_f* cases; src/lib/testsuite/test_raft.c: `test_leader_lost_keeps_voted_for`, `test_leader_lost_same_term_vote_from_other_denied`, `test_check_and_start_after_leader_lost_same_term_starts`, `test_check_and_start_skips_when_not_follower`. Written ahead of each change in the 2026-10-03 fix sessions (RED/GREEN per the session log; RED not re-run on 2026-10-04).
- Suites: `xce-build.sh all` asan, 127/127 meson suites OK + Go tests ok (2026-10-04, together with bug-1919).
- Live: 2026-10-03: `/xce-test-ha` chaotic 6 and loaded 8 rounds PASS; the sequential 20-round run failed on bug-1919 (fence), with a clean single-decision election timeline. 2026-10-04: 4x sequential 20 rounds, 3 PASS, 1 FAIL on bug-1920 (catch-up stream stall after a log-mismatch NACK; under investigation, relation to this change not established). Across the four 2026-10-04 runs, 94 PreVote rounds were decided WON and none twice (one "PreVote round N WON" per pid+round+term).
## References
- Related: none