lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): bug-1435 raft_lock serializes concurrent raft_state_t access

Browse source 
raft-drive-async's default-ON pool dispatch lets messages from different
peer endpoints (two vote replies, or a vote reply racing a vote request)
run concurrently on different vtask pool workers. raft_state_t's fields
(votes_granted, replies_received, election_decided, term, role) are plain
non-atomic — the raft library assumes a single-threaded caller. Two
concurrent raft_on_vote_reply() calls could race: one reaches quorum and
promotes to PRIMARY while the other, reading pre-increment vote count,
independently satisfies "all-replied, no quorum" and fires
on_become_follower for the same term, silently reverting the promotion —
every request afterward correctly (but wrongly) rejected as STANDBY.

Add ha->raft_lock (PTHREAD_MUTEX_RECURSIVE — a WON/quorum outcome can
legitimately re-enter the state machine synchronously on the same thread,
e.g. PreVote WON -> real election) around every raft_state_t entry point
on the election/vote path. Verified via 40-round sequential and 40-round
chaotic HA soak, zero committed-job loss, all 3 nodes led.
This commit is contained in:
Rongsong Shen 2026-07-13 07:18:44 +00:00
parent 976a6c956b
commit 510b566bbf
3 changed files with 121 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

15
src/controller/cluster.c
View file

@ -468,8 +468,15 @@ static int myreq_handler(vtask_t* task)
uint64_t reply_term = 0;
int granted = 0;
if (freshness_ok && from_idx >= 0) {
/* bug-1435: serialize against concurrent raft_state_t access —
* this node's own vote-reply/election processing (a different
* peer connection, dispatched on a different pool worker) can
* race this vote REQUEST on the same ha->raft (see ha.h
* raft_lock doc). */
pthread_mutex_lock(&c->ha.raft_lock);
int r = raft_on_vote_request(&c->ha.raft, from_idx, cand_term,
cand_llt, cand_lli, &reply_term);
pthread_mutex_unlock(&c->ha.raft_lock);
granted = (r == RAFT_GRANT) ? 1 : 0;
} else {
/* Freshness deny or unknown candidate: read current term for reply */
@ -565,9 +572,15 @@ static int myreq_handler(vtask_t* task)
deny_reason = "unknown_candidate";
} else {
/* raft_on_pre_vote_request is pure: stale-term + eligibility gate,
* no term/voted_for/role mutation, no persist. */
* no term/voted_for/role mutation, no persist. Still reads
* raft_state_t's term/role — bug-1435: take raft_lock so this
* read cannot observe a torn (mid-transition) snapshot while a
* concurrent vote-reply/election-start call on another pool
* worker is mutating the same fields (see ha.h raft_lock doc). */
pthread_mutex_lock(&c->ha.raft_lock);
int r = raft_on_pre_vote_request(&c->ha.raft, from_idx, cand_term,
cand_llt, cand_lli);
pthread_mutex_unlock(&c->ha.raft_lock);
if (r == RAFT_GRANT) {
granted = 1;
} else {