lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): cold start elects in ~1.7 s, not ~10 s; ctrlstatus appliedLsn on the primary (bug-1879, bug-1878)

Browse source 
bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its
term-1 election before its peers ran; connect failures count as "no"
votes (bug-1423), so it lost at once and became a follower with no
leader. ctrl-2/ctrl-3 hold no config and are voters only (by design),
so nothing retried until ctrl-1's 10 s standby liveness timeout: every
cold start had no primary for ~10 s.
- ha_note_leaderless() marks a config-holding node that becomes a
  follower with no leader known; the liveness poll then retries the
  election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to
  heartbeat_timeout_ms. Primary contact or becoming leader clears it.
- ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity
  then only holds the watchdog's arm time, and the failover gates took
  it for a live primary.
Live cold start: primary 1.7 s after the first node started (was ~10 s).

bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a
standby-only counter a primary never advances, so every primary showed
"appliedLsn: 0". A primary now reports persist->last_applied_lsn. The
controller-status design doc's "already correct in every role" note is
corrected.

Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c
(RED: expected 4900, was 0); controller suite 1256/0. HA soaks:
sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero
job loss).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-28 07:35:45 +00:00
parent 16b422e303
commit aa7fc65a36
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
6 changed files with 260 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

15
src/controller/ha.h
View file

@ -134,6 +134,9 @@ typedef struct {
#define HA_DEFAULT_HEARTBEAT_INTERVAL_MS 3000
#define HA_DEFAULT_HEARTBEAT_TIMEOUT_MS 10000
#define HA_DEFAULT_ELECTION_TIMEOUT_MS 5000
/* bug-1879: first election retry of a leaderless config-holding node;
* doubles per attempt up to heartbeat_timeout_ms */
#define HA_LEADERLESS_RETRY_MS 1000
#define HA_DEFAULT_PROBE_CONNECT_TIMEOUT_MS 500
/* Log the first connect failure per peer, then every Nth attempt. */
#define HA_CONNECT_LOG_THROTTLE 10
@ -247,6 +250,16 @@ typedef struct {
void *raft_election_timeout_ctx; /* election_timeout_ctx_t* for cancel (task 6.2) */
/* ── New fields added by ha-heartbeat-liveness spec (Task 1.1) ── */
time_t last_activity; /* CLOCK_MONOTONIC seconds of last primary message; 0 = none yet */
/* bug-1879: set (ha_note_leaderless) when this node, able to lead (it
* holds a config), became a follower without knowing any leader -- e.g.
* its first-boot election lost because its peers were not up yet. The
* liveness poll then retries the election after leaderless_retry_ms
* (measured from leaderless_since_ms, CLOCK_MONOTONIC ms; doubling up to
* heartbeat_timeout_ms) instead of the full heartbeat_timeout_ms. The
* first primary contact (ha_on_heartbeat_received) clears it. */
_Atomic int leaderless;
uint32_t leaderless_retry_ms;
uint64_t leaderless_since_ms;
/* ── New fields added by ha-repl-hardening spec (Task 1.4) ──────── */
uint64_t standby_reported_tag[MAX_HA_CANDIDATES]; /* Last config_tag reported by each standby */
uint64_t standby_reported_lsn[MAX_HA_CANDIDATES]; /* Last applied_lsn reported by each standby */
@ -956,6 +969,8 @@ int ha_schedule_phase2_retry(cluster_t *c);
/* bug-1876: cluster_init_phase2()'s "cluster_config not pushed yet" result
* (-EAGAIN) -- the normal cold-boot wait of a standby, retried, not a
* failure; callers log it at INFO, everything else at WARN. */
void ha_note_leaderless(ha_state_t *ha, cluster_t *c, int leader_idx);
static inline int ha_phase2_rc_is_waiting(int rc)
{
return rc == -EAGAIN;