lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha-single-leader-role,ha-repl-hardening): promotion must register standbys with the WAL replicator

Browse source 
Live symptom: every write (jobsub) on a healthy 3-node cluster returned
E_XCE_NOT_READY (10013) — the primary had no write quorum and committed_lsn was
stuck at 0, even though both standbys were connected, heartbeating, streamed, and
the primary RECEIVED their WAL ACKs.

Root cause: raft_cb_become_leader registers a surviving vote endpoint with
repl_add_standby only when ectx->votes[i].peer != NULL, but raft_cb_send_vote_request
set votes[i].ep and never votes[i].peer. So at promotion the guard failed, the
standbys were added to ha->standby_endpoints (streaming) but NOT to the replicator
array c->persist.repl.standbys[] (commit/quorum). Consequently repl_on_ack could
not find the ack's ep (slot_idx=-1) -> commit_on_ack never ran -> committed_lsn
never advanced; majority_candidate/repl_active_standby_count saw 0 standbys ->
no write quorum -> the dispatch gate rejected every write with -XCE_ERR_NOT_READY.
(Streaming/persistence worked, which masked it: reslist/workerlist succeed, only
writes fail.)

Fix: raft_cb_send_vote_request now also sets ectx->votes[votes_idx].peer = peer
(the &candidates[peer_idx] pointer already computed for the connect) at the same
success site as .ep. peer points into the persistent myconf candidates[];
repl_add_standby copies host/port, so no lifetime dependency. Promotion now
registers each surviving vote ep -> standbys enter the replicator -> WAL ACKs
advance committed_lsn -> own-term gate opens -> write quorum established.

TDD: drive the real raft_cb_become_leader over a real persist/repl/commit fixture
and assert repl_active_standby_count post-promotion == N when votes[].peer is set,
== 0 when NULL (the bug). Full controller suite green under ASAN; election/commit/
repl/wire untouched apart from the one-line .peer assignment + a test seam.
This commit is contained in:
Rongsong Shen 2026-06-14 14:57:44 +00:00
parent 4c5b27cb70
commit c4fd3e1c8f
3 changed files with 198 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

13
src/controller/ha.h
View file

@ -1008,6 +1008,19 @@ extern void (*ha_fatal_hook)(const char *msg);
*/
void raft_cb_become_follower_with_cluster_for_test(cluster_t *c, uint64_t term);
/*
* raft_cb_become_leader_with_cluster_for_test: drive the leader promotion path
* with a LIVE cluster pointer (fix-promotion-standby-registration). Unlike
* raft_cb_become_leader_for_test (which forces raft_c = NULL and so skips the
* c-dependent standby-registration branch entirely), this shim sets
* ha->raft_c = c first so the promotion-time repl_add_standby loop — which
* registers each surviving vote endpoint (votes[i].ep != NULL) carrying a
* peer pointer (votes[i].peer != NULL) into the WAL replicator — is exercised
* against a real persist/repl/commit fixture. Symmetric with
* raft_cb_become_follower_with_cluster_for_test.
*/
void raft_cb_become_leader_with_cluster_for_test(cluster_t *c, uint64_t term);
/*
* ha_walack_continuation_for_test: test-only entry point for the ACK-driven
* catch-up continuation logic (ha-convergence-recovery task 4.x, Req 5.1, 5.2,