lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha-single-leader-role,ha-repl-hardening): promotion must register standbys with the WAL replicator

Browse source 
Live symptom: every write (jobsub) on a healthy 3-node cluster returned
E_XCE_NOT_READY (10013) — the primary had no write quorum and committed_lsn was
stuck at 0, even though both standbys were connected, heartbeating, streamed, and
the primary RECEIVED their WAL ACKs.

Root cause: raft_cb_become_leader registers a surviving vote endpoint with
repl_add_standby only when ectx->votes[i].peer != NULL, but raft_cb_send_vote_request
set votes[i].ep and never votes[i].peer. So at promotion the guard failed, the
standbys were added to ha->standby_endpoints (streaming) but NOT to the replicator
array c->persist.repl.standbys[] (commit/quorum). Consequently repl_on_ack could
not find the ack's ep (slot_idx=-1) -> commit_on_ack never ran -> committed_lsn
never advanced; majority_candidate/repl_active_standby_count saw 0 standbys ->
no write quorum -> the dispatch gate rejected every write with -XCE_ERR_NOT_READY.
(Streaming/persistence worked, which masked it: reslist/workerlist succeed, only
writes fail.)

Fix: raft_cb_send_vote_request now also sets ectx->votes[votes_idx].peer = peer
(the &candidates[peer_idx] pointer already computed for the connect) at the same
success site as .ep. peer points into the persistent myconf candidates[];
repl_add_standby copies host/port, so no lifetime dependency. Promotion now
registers each surviving vote ep -> standbys enter the replicator -> WAL ACKs
advance committed_lsn -> own-term gate opens -> write quorum established.

TDD: drive the real raft_cb_become_leader over a real persist/repl/commit fixture
and assert repl_active_standby_count post-promotion == N when votes[].peer is set,
== 0 when NULL (the bug). Full controller suite green under ASAN; election/commit/
repl/wire untouched apart from the one-line .peer assignment + a test seam.
This commit is contained in:
Rongsong Shen 2026-06-14 14:57:44 +00:00
parent 4c5b27cb70
commit c4fd3e1c8f
3 changed files with 198 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

32
src/controller/ha.c
View file

@ -187,8 +187,18 @@ static int raft_cb_send_vote_request(void *ctx, int peer_idx, uint64_t term)
return 1;
}
/* Track endpoint for promotion/close after election result */
ectx->votes[votes_idx].ep = ep;
/* Track endpoint for promotion/close after election result.
* Also record the candidate pointer: raft_cb_become_leader uses
* votes[i].peer->hostname / ->port to register the surviving vote
* endpoint with the WAL replicator (repl_add_standby). Without this,
* the promotion registration block (guarded by votes[i].peer != NULL)
* is skipped — the standbys never enter c->persist.repl.standbys[],
* commit/quorum see zero standbys, and every write fails NOT_READY.
* `peer` points into c->myconf->peers_cfg.candidates[] (persistent;
* outlives the election — repl_add_standby copies host/port).
* (fix-promotion-standby-registration, Req 1.1) */
ectx->votes[votes_idx].ep = ep;
ectx->votes[votes_idx].peer = peer;
return 0;
}
@ -3194,6 +3204,24 @@ void raft_cb_become_follower_with_cluster_for_test(cluster_t *c, uint64_t term)
c->ha.raft_c = c;
raft_cb_become_follower((void *)&c->ha, term, -1);
}
/*
* raft_cb_become_leader_with_cluster_for_test (fix-promotion-standby-registration):
* drive the leader promotion path with a LIVE cluster pointer so the
* cluster-dependent standby-registration branch (repl_add_standby for each
* surviving vote endpoint whose votes[i].peer is set) is exercised against a
* real persist/repl/commit/delaymgr fixture. Sets ha->raft_c = c so the
* become-leader callback sees a real cluster (unlike the NULL-cluster shim
* raft_cb_become_leader_for_test above, which suppresses the c-dependent
* registration block entirely). raft_cb_become_leader is file-static, so this
* seam must live here in ha.c. Symmetric with
* raft_cb_become_follower_with_cluster_for_test.
*/
void raft_cb_become_leader_with_cluster_for_test(cluster_t *c, uint64_t term)
{
c->ha.raft_c = c;
raft_cb_become_leader((void *)&c->ha, term);
}
#endif /* XCE_TESTING */
/* ── Task 2.4: vote_ep_err_handler ─────────────────────────────────────────