lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha-bootstrap): election retry loop until quorum achieved

Browse source 
Replace single 2-round election attempt with an infinite retry loop:
  1. ha_wait_for_heartbeat_or_timeout() — catch existing primary's heartbeat
     (→ STANDBY immediately) or timeout and proceed to election
  2. am_i_primary_quorum() — run 2-round election
     (→ PRIMARY if won, otherwise loop back to step 1)

Nodes now retry indefinitely until one wins a quorum. On split vote
or no-peer-reachable scenarios the node loops back; on the next
heartbeat wait, it either catches the winning primary's heartbeat
(→ STANDBY) or times out and tries again with a higher term and
randomized jitter (Raft's split-vote resolution mechanism).

This is the core Raft liveness property: as long as a majority of
nodes are reachable, the cluster eventually elects a leader.
This commit is contained in:
Rongsong Shen 2026-05-30 02:41:06 +00:00
parent 8468a83aca
commit fcfb82f7c7
1 changed files with 58 additions and 53 deletions
Download patch file Download diff file Expand all files Collapse all files

111
src/controller/myapp.c
View file

@ -113,72 +113,74 @@ int myapp_run(myconf_t* myconf)
return rc;
}
/* Req 1.1: wait for an incoming heartbeat before starting election.
* If a primary already exists it will send heartbeats; receiving one means
* this node should become STANDBY immediately (skip election).
* If no heartbeat arrives within election_timeout_ms + jitter, proceed to
* election — no primary is reachable and this node should compete. */
int skip_election = 0;
{
ctrl_role_t hb_role = ha_wait_for_heartbeat_or_timeout(&c.ha);
if (hb_role == CTRL_STANDBY) {
/* A primary is already running — skip election, go straight to standby */
INFO("HA: heartbeat received — existing primary detected, becoming STANDBY");
skip_election = 1;
}
/* CTRL_ELECTING: no heartbeat received, proceed to election */
/* Election retry loop: keep running until this node becomes PRIMARY or
* confirms a primary exists (via heartbeat or election loss).
*
* Iteration:
* 1. Wait for heartbeat (election_timeout_ms + jitter).
* - Heartbeat received → existing primary detected → STANDBY, exit loop.
* - Timeout → no primary visible, proceed to election.
* 2. Run election (2 rounds with jitter).
* - Won PRIMARY → exit loop as primary.
* - Lost (split vote or peer won) → loop back to step 1.
* The next heartbeat wait will catch the winner's heartbeat if one exists,
* or fire after election_timeout_ms if it was a split vote.
*/
ctrl_role_t final_role = CTRL_ELECTING;
/* Build peer pointer array once (stable across iterations) */
const xce_cluster_config_t *pcfg = &myconf->peers_cfg;
int n_peers = (pcfg->candidates_count > 0) ? pcfg->candidates_count : 1;
xce_candidate_t *peer_ptrs[MAX_HA_CANDIDATES];
int n_other = 0;
for (int i = 0; i < pcfg->candidates_count; i++) {
if (i != c.ha.my_priority)
peer_ptrs[n_other++] = (xce_candidate_t *)&pcfg->candidates[i];
}
if (!skip_election) {
/* Election: quorum-based role determination (Req 2.1, 2.9). */
const xce_cluster_config_t *pcfg = &myconf->peers_cfg;
int n_peers = (pcfg->candidates_count > 0) ? pcfg->candidates_count : 1;
xce_candidate_t *peer_ptrs[MAX_HA_CANDIDATES];
int n_other = 0;
for (int i = 0; i < pcfg->candidates_count; i++) {
if (i != c.ha.my_priority)
peer_ptrs[n_other++] = (xce_candidate_t *)&pcfg->candidates[i];
for (;;) {
/* Step 1: wait for heartbeat from an existing primary */
ctrl_role_t hb_role = ha_wait_for_heartbeat_or_timeout(&c.ha);
if (hb_role == CTRL_STANDBY) {
INFO("HA: heartbeat received — existing primary detected, becoming STANDBY");
final_role = CTRL_STANDBY;
break;
}
/* Step 2: no heartbeat — run election */
uint64_t my_tag = myconf->config_version_tag;
uint64_t my_lsn = (uint64_t)atomic_load_explicit(&c.ha.applied_lsn,
memory_order_acquire);
ctrl_role_t elected = am_i_primary_quorum(
&c.ha, n_peers, (n_other > 0) ? peer_ptrs : NULL, my_tag, my_lsn);
if (elected == CTRL_PRIMARY) {
rc = load_primary_config(&c);
if (rc != 0) {
cluster_fini(&c);
myconf_fini(myconf);
return rc;
}
atomic_store_explicit(&c.ha.role, (int32_t)CTRL_PRIMARY,
memory_order_release);
} else {
if (c.ha.defer_config_fetch) {
const xce_cluster_config_t *p2 = &myconf->peers_cfg;
if (p2->candidates_count > 0)
INFO("HA bootstrap: deferred config fetch from winner %s:%u",
p2->candidates[0].hostname, (unsigned)p2->candidates[0].port);
c.ha.defer_config_fetch = false;
}
if (myconf->config_version_tag == 0) {
rc = fetch_config_from_primary(&c);
if (rc != 0) {
cluster_fini(&c);
myconf_fini(myconf);
return rc;
}
} else {
INFO("Standby has local config (tag=%016llx) — skipping fetch from primary",
(unsigned long long)myconf->config_version_tag);
}
final_role = CTRL_PRIMARY;
break;
}
/* Lost election — loop back; next heartbeat wait will resolve role */
INFO("HA: election inconclusive — retrying after heartbeat wait");
}
/* Post-loop: apply role */
if (final_role == CTRL_PRIMARY) {
rc = load_primary_config(&c);
if (rc != 0) {
cluster_fini(&c);
myconf_fini(myconf);
return rc;
}
atomic_store_explicit(&c.ha.role, (int32_t)CTRL_PRIMARY,
memory_order_release);
} else {
/* Heartbeat-detected STANDBY: fetch config if not already loaded. */
/* STANDBY: fetch config from primary if not already loaded */
if (c.ha.defer_config_fetch) {
if (pcfg->candidates_count > 0)
INFO("HA bootstrap: deferred config fetch from winner %s:%u",
pcfg->candidates[0].hostname, (unsigned)pcfg->candidates[0].port);
c.ha.defer_config_fetch = false;
}
if (myconf->config_version_tag == 0) {
rc = fetch_config_from_primary(&c);
if (rc != 0) {
@ -186,6 +188,9 @@ int myapp_run(myconf_t* myconf)
myconf_fini(myconf);
return rc;
}
} else {
INFO("Standby has local config (tag=%016llx) — skipping fetch from primary",
(unsigned long long)myconf->config_version_tag);
}
}