lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha-bootstrap): election retry loop until quorum achieved

Browse source 
Replace single 2-round election attempt with an infinite retry loop:
  1. ha_wait_for_heartbeat_or_timeout() — catch existing primary's heartbeat
     (→ STANDBY immediately) or timeout and proceed to election
  2. am_i_primary_quorum() — run 2-round election
     (→ PRIMARY if won, otherwise loop back to step 1)

Nodes now retry indefinitely until one wins a quorum. On split vote
or no-peer-reachable scenarios the node loops back; on the next
heartbeat wait, it either catches the winning primary's heartbeat
(→ STANDBY) or times out and tries again with a higher term and
randomized jitter (Raft's split-vote resolution mechanism).

This is the core Raft liveness property: as long as a majority of
nodes are reachable, the cluster eventually elects a leader.
This commit is contained in:
Rongsong Shen 2026-05-30 02:41:06 +00:00
parent 8468a83aca
commit fcfb82f7c7
1 changed files with 58 additions and 53 deletions
Download patch file Download diff file Expand all files Collapse all files

111
src/controller/myapp.c
View file

@ -113,72 +113,74 @@ int myapp_run(myconf_t* myconf)
return rc; return rc;
} }
/* Req 1.1: wait for an incoming heartbeat before starting election. /* Election retry loop: keep running until this node becomes PRIMARY or
* If a primary already exists it will send heartbeats; receiving one means * confirms a primary exists (via heartbeat or election loss).
* this node should become STANDBY immediately (skip election). *
* If no heartbeat arrives within election_timeout_ms + jitter, proceed to * Iteration:
* election — no primary is reachable and this node should compete. */ * 1. Wait for heartbeat (election_timeout_ms + jitter).
int skip_election = 0; * - Heartbeat received → existing primary detected → STANDBY, exit loop.
{ * - Timeout → no primary visible, proceed to election.
ctrl_role_t hb_role = ha_wait_for_heartbeat_or_timeout(&c.ha); * 2. Run election (2 rounds with jitter).
if (hb_role == CTRL_STANDBY) { * - Won PRIMARY → exit loop as primary.
/* A primary is already running — skip election, go straight to standby */ * - Lost (split vote or peer won) → loop back to step 1.
INFO("HA: heartbeat received — existing primary detected, becoming STANDBY"); * The next heartbeat wait will catch the winner's heartbeat if one exists,
skip_election = 1; * or fire after election_timeout_ms if it was a split vote.
} */
/* CTRL_ELECTING: no heartbeat received, proceed to election */ ctrl_role_t final_role = CTRL_ELECTING;
/* Build peer pointer array once (stable across iterations) */
const xce_cluster_config_t *pcfg = &myconf->peers_cfg;
int n_peers = (pcfg->candidates_count > 0) ? pcfg->candidates_count : 1;
xce_candidate_t *peer_ptrs[MAX_HA_CANDIDATES];
int n_other = 0;
for (int i = 0; i < pcfg->candidates_count; i++) {
if (i != c.ha.my_priority)
peer_ptrs[n_other++] = (xce_candidate_t *)&pcfg->candidates[i];
} }
if (!skip_election) { for (;;) {
/* Election: quorum-based role determination (Req 2.1, 2.9). */ /* Step 1: wait for heartbeat from an existing primary */
const xce_cluster_config_t *pcfg = &myconf->peers_cfg; ctrl_role_t hb_role = ha_wait_for_heartbeat_or_timeout(&c.ha);
int n_peers = (pcfg->candidates_count > 0) ? pcfg->candidates_count : 1; if (hb_role == CTRL_STANDBY) {
INFO("HA: heartbeat received — existing primary detected, becoming STANDBY");
xce_candidate_t *peer_ptrs[MAX_HA_CANDIDATES]; final_role = CTRL_STANDBY;
int n_other = 0; break;
for (int i = 0; i < pcfg->candidates_count; i++) {
if (i != c.ha.my_priority)
peer_ptrs[n_other++] = (xce_candidate_t *)&pcfg->candidates[i];
} }
/* Step 2: no heartbeat — run election */
uint64_t my_tag = myconf->config_version_tag; uint64_t my_tag = myconf->config_version_tag;
uint64_t my_lsn = (uint64_t)atomic_load_explicit(&c.ha.applied_lsn, uint64_t my_lsn = (uint64_t)atomic_load_explicit(&c.ha.applied_lsn,
memory_order_acquire); memory_order_acquire);
ctrl_role_t elected = am_i_primary_quorum( ctrl_role_t elected = am_i_primary_quorum(
&c.ha, n_peers, (n_other > 0) ? peer_ptrs : NULL, my_tag, my_lsn); &c.ha, n_peers, (n_other > 0) ? peer_ptrs : NULL, my_tag, my_lsn);
if (elected == CTRL_PRIMARY) { if (elected == CTRL_PRIMARY) {
rc = load_primary_config(&c); final_role = CTRL_PRIMARY;
if (rc != 0) { break;
cluster_fini(&c);
myconf_fini(myconf);
return rc;
}
atomic_store_explicit(&c.ha.role, (int32_t)CTRL_PRIMARY,
memory_order_release);
} else {
if (c.ha.defer_config_fetch) {
const xce_cluster_config_t *p2 = &myconf->peers_cfg;
if (p2->candidates_count > 0)
INFO("HA bootstrap: deferred config fetch from winner %s:%u",
p2->candidates[0].hostname, (unsigned)p2->candidates[0].port);
c.ha.defer_config_fetch = false;
}
if (myconf->config_version_tag == 0) {
rc = fetch_config_from_primary(&c);
if (rc != 0) {
cluster_fini(&c);
myconf_fini(myconf);
return rc;
}
} else {
INFO("Standby has local config (tag=%016llx) — skipping fetch from primary",
(unsigned long long)myconf->config_version_tag);
}
} }
/* Lost election — loop back; next heartbeat wait will resolve role */
INFO("HA: election inconclusive — retrying after heartbeat wait");
}
/* Post-loop: apply role */
if (final_role == CTRL_PRIMARY) {
rc = load_primary_config(&c);
if (rc != 0) {
cluster_fini(&c);
myconf_fini(myconf);
return rc;
}
atomic_store_explicit(&c.ha.role, (int32_t)CTRL_PRIMARY,
memory_order_release);
} else { } else {
/* Heartbeat-detected STANDBY: fetch config if not already loaded. */ /* STANDBY: fetch config from primary if not already loaded */
if (c.ha.defer_config_fetch) {
if (pcfg->candidates_count > 0)
INFO("HA bootstrap: deferred config fetch from winner %s:%u",
pcfg->candidates[0].hostname, (unsigned)pcfg->candidates[0].port);
c.ha.defer_config_fetch = false;
}
if (myconf->config_version_tag == 0) { if (myconf->config_version_tag == 0) {
rc = fetch_config_from_primary(&c); rc = fetch_config_from_primary(&c);
if (rc != 0) { if (rc != 0) {
@ -186,6 +188,9 @@ int myapp_run(myconf_t* myconf)
myconf_fini(myconf); myconf_fini(myconf);
return rc; return rc;
} }
} else {
INFO("Standby has local config (tag=%016llx) — skipping fetch from primary",
(unsigned long long)myconf->config_version_tag);
} }
} }