lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): cold start elects in ~1.7 s, not ~10 s; ctrlstatus appliedLsn on the primary (bug-1879, bug-1878)

Browse source 
bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its
term-1 election before its peers ran; connect failures count as "no"
votes (bug-1423), so it lost at once and became a follower with no
leader. ctrl-2/ctrl-3 hold no config and are voters only (by design),
so nothing retried until ctrl-1's 10 s standby liveness timeout: every
cold start had no primary for ~10 s.
- ha_note_leaderless() marks a config-holding node that becomes a
  follower with no leader known; the liveness poll then retries the
  election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to
  heartbeat_timeout_ms. Primary contact or becoming leader clears it.
- ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity
  then only holds the watchdog's arm time, and the failover gates took
  it for a live primary.
Live cold start: primary 1.7 s after the first node started (was ~10 s).

bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a
standby-only counter a primary never advances, so every primary showed
"appliedLsn: 0". A primary now reports persist->last_applied_lsn. The
controller-status design doc's "already correct in every role" note is
corrected.

Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c
(RED: expected 4900, was 0); controller suite 1256/0. HA soaks:
sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero
job loss).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-28 07:35:45 +00:00
parent 16b422e303
commit aa7fc65a36
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
6 changed files with 260 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

65
src/controller/ha.c
View file

@ -1621,6 +1621,9 @@ loud_fence_no_timer:
static void raft_cb_become_leader(void *ctx, uint64_t term)
{
ha_state_t *ha = (ha_state_t *)ctx;
/* bug-1879: a leader exists (us) -- no leaderless retry pending */
atomic_store_explicit(&ha->leaderless, 0, memory_order_release);
cluster_t *c = ha->raft_c;
/* Cancel any pending election timeout */
@ -2144,12 +2147,34 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
* calls cluster_init_phase2 on first boot (standbys also need it to
* receive config replication push from the new primary).
*/
static uint64_t ha_monotonic_ms(void)
{
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return (uint64_t)ts.tv_sec * 1000u + (uint64_t)ts.tv_nsec / 1000000u;
}
/* bug-1879: see ha.h's `leaderless`. Only a node that can lead (holds a
* config) and knows no leader is marked; a config-less node is a voter only
* (ha_start_election_async) and keeps the plain liveness timeout. */
void ha_note_leaderless(ha_state_t *ha, cluster_t *c, int leader_idx)
{
if (!ha || leader_idx >= 0 || !c || !c->myconf
|| c->myconf->config_version_tag == 0)
return;
if (!atomic_exchange_explicit(&ha->leaderless, 1, memory_order_acq_rel))
ha->leaderless_retry_ms = HA_LEADERLESS_RETRY_MS;
}
static void raft_cb_become_follower(void *ctx, uint64_t term, int leader_idx)
{
(void)leader_idx;
ha_state_t *ha = (ha_state_t *)ctx;
cluster_t *c = ha->raft_c;
/* bug-1879: a lost election with no leader known is retried soon, not
* after heartbeat_timeout_ms (armed below) */
ha_note_leaderless(ha, c, leader_idx);
/* Cancel any pending election timeout */
raft_cb_cancel_timeout(ctx);
@ -5423,6 +5448,10 @@ int ha_heartbeat_is_fresh(ha_state_t *ha)
time_t last = ha->last_activity; /* time_t load is atomic on LP64 */
if (last == 0)
return 0; /* no primary message seen yet */
/* bug-1879: while leaderless no primary has been heard from at all --
* last_activity then only holds the watchdog's arm time */
if (atomic_load_explicit(&ha->leaderless, memory_order_acquire))
return 0;
struct timespec now;
clock_gettime(CLOCK_MONOTONIC, &now);
long ms = (long)(now.tv_sec - last) * 1000L;
@ -8993,6 +9022,22 @@ static int liveness_poll_fn(void *arg)
time_t elapsed = (last > 0) ? (now - last) : 0;
uint32_t timeout_s = ha->config.heartbeat_timeout_ms / 1000;
/* bug-1879: a leaderless config-holding node retries the election after
* its (doubling) retry delay instead of the full heartbeat timeout */
if (atomic_load_explicit(&ha->leaderless, memory_order_acquire)) {
uint64_t waited = ha_monotonic_ms() - ha->leaderless_since_ms;
if (waited >= ha->leaderless_retry_ms) {
INFO("HA: no primary known after %lums — retrying the election",
(unsigned long)waited);
ha->leaderless_retry_ms *= 2;
if (ha->leaderless_retry_ms > ha->config.heartbeat_timeout_ms)
ha->leaderless_retry_ms = ha->config.heartbeat_timeout_ms;
ha_dispatch_rdhup_failover(ha, c);
return 0; /* failover (PreVote lost -> re-arm) handles the rest */
}
}
if (last > 0 && (uint32_t)elapsed >= timeout_s) {
/* Silence exceeded timeout — trigger failover */
WARN("HA: liveness poll: no primary activity for %lus (timeout=%us) → failover",
@ -9006,6 +9051,10 @@ static int liveness_poll_fn(void *arg)
if (c && c->delaymgr.running) {
uint32_t poll_ms = ha->config.heartbeat_interval_ms / 2;
if (poll_ms < 100) poll_ms = 100;
/* bug-1879: poll no slower than a pending leaderless retry */
if (atomic_load_explicit(&ha->leaderless, memory_order_acquire)
&& ha->leaderless_retry_ms < poll_ms)
poll_ms = ha->leaderless_retry_ms;
delay_task_ms(&c->delaymgr, (int)poll_ms, 1,
(delay_task_entry_fn_t)liveness_poll_fn, ctx);
}
@ -9079,6 +9128,15 @@ void ha_arm_heartbeat_timeout_vtask(ha_state_t *ha, cluster_t *c)
uint32_t poll_ms = ha->config.heartbeat_interval_ms / 2;
if (poll_ms < 100) poll_ms = 100;
if (atomic_load_explicit(&ha->leaderless, memory_order_acquire)) {
/* bug-1879: no leader known -- poll at the retry delay */
ha->leaderless_since_ms = ha_monotonic_ms();
if (ha->leaderless_retry_ms < poll_ms)
poll_ms = ha->leaderless_retry_ms;
INFO("HA: no primary known yet — election retry in %ums",
ha->leaderless_retry_ms);
}
INFO("HA: standby liveness poll armed (timeout=%ums, poll=%ums)",
ha->config.heartbeat_timeout_ms, poll_ms);
@ -9138,6 +9196,11 @@ void ha_on_heartbeat_received(ha_state_t *ha)
ha->last_activity = ts.tv_sec;
tlog(TLOG_DEBUG3, "HA: heartbeat received — last_activity updated to %ld", (long)ts.tv_sec);
/* bug-1879: a primary exists -- back to the plain liveness timeout */
if (atomic_exchange_explicit(&ha->leaderless, 0, memory_order_acq_rel))
INFO("HA: primary contact — leaderless election retry stopped");
ha->leaderless_retry_ms = HA_LEADERLESS_RETRY_MS;
/* ha-prevote task 4.2 (Req 4.2): a fresh primary heartbeat means the leader
* is alive — a PreVote round started by a transient blip must be ABANDONED.
* Cancel the in-progress round (clear the raft round state + the controller