lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): bug-1435 raft_lock serializes concurrent raft_state_t access

Browse source 
raft-drive-async's default-ON pool dispatch lets messages from different
peer endpoints (two vote replies, or a vote reply racing a vote request)
run concurrently on different vtask pool workers. raft_state_t's fields
(votes_granted, replies_received, election_decided, term, role) are plain
non-atomic — the raft library assumes a single-threaded caller. Two
concurrent raft_on_vote_reply() calls could race: one reaches quorum and
promotes to PRIMARY while the other, reading pre-increment vote count,
independently satisfies "all-replied, no quorum" and fires
on_become_follower for the same term, silently reverting the promotion —
every request afterward correctly (but wrongly) rejected as STANDBY.

Add ha->raft_lock (PTHREAD_MUTEX_RECURSIVE — a WON/quorum outcome can
legitimately re-enter the state machine synchronously on the same thread,
e.g. PreVote WON -> real election) around every raft_state_t entry point
on the election/vote path. Verified via 40-round sequential and 40-round
chaotic HA soak, zero committed-job loss, all 3 nodes led.
This commit is contained in:
Rongsong Shen 2026-07-13 07:18:44 +00:00
parent 976a6c956b
commit 510b566bbf
3 changed files with 121 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

28
src/controller/ha.h
View file

@ -196,6 +196,34 @@ typedef struct {
int n_peers; /* total cluster size (including self), matches raft convention */
/* ── New fields added by raft-library spec (Task 6.1) ──────── */
raft_state_t raft; /* Raft consensus state (wired in task 6.2) */
/* bug-1435 (raft-drive-async concurrency gap): raft_state_t's own fields
* (term/role/votes_granted/replies_received/election_decided/vote_replies[]
* etc.) are plain, non-atomic and have NO protection of their own — the
* Raft library assumes a single-threaded/serialized caller (the classic
* "single event-loop" consensus model). But raft-drive-async's default-ON
* pool dispatch (peer_ctrl_dispatch_to_pool, VTASK_LIVENESS) lets messages
* from DIFFERENT peer endpoints (e.g. two RequestVoteReply's from two
* different voters, or a vote reply racing a vote request) run
* CONCURRENTLY on different vtask pool workers — per-endpoint
* serialization (bind_vtask) only orders messages on the SAME endpoint,
* not across endpoints. Two concurrent raft_on_vote_reply() calls can
* race on votes_granted/replies_received/election_decided: e.g. one
* worker's grant reaches quorum and fires on_become_leader (role→PRIMARY)
* while a second worker, concurrently processing another peer's reply,
* reads pre-increment votes_granted and independently satisfies the
* "all-replied, no quorum" branch, firing on_become_follower (role→
* STANDBY) an instant later for the SAME term — silently undoing the
* promotion with no further election. raft_lock serializes every mutating
* entry point into `raft` (election/vote-reply/vote-request path only —
* see call sites) so at most one thread evaluates the state machine at a
* time, matching the raft library's single-threaded design assumption.
* NOT the same lock as `lock` above (which protects unrelated fields) —
* kept separate so the two never nest in conflicting order. Initialized
* PTHREAD_MUTEX_RECURSIVE (ha_init): a WON/quorum outcome can legitimately
* re-enter the state machine synchronously on the SAME thread (PreVote WON
* -> real election, within one call chain) — that nesting must not
* self-deadlock, while a genuinely different thread still fully blocks. */
pthread_mutex_t raft_lock;
/* ── New fields added by raft-library spec (Task 6.2) ──────── */
cluster_t *raft_c; /* cluster_t* set by ha_start_election_async; used by raft callbacks */
void *raft_election_timeout_ctx; /* election_timeout_ctx_t* for cancel (task 6.2) */