lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): cold start elects in ~1.7 s, not ~10 s; ctrlstatus appliedLsn on the primary (bug-1879, bug-1878)

Browse source 
bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its
term-1 election before its peers ran; connect failures count as "no"
votes (bug-1423), so it lost at once and became a follower with no
leader. ctrl-2/ctrl-3 hold no config and are voters only (by design),
so nothing retried until ctrl-1's 10 s standby liveness timeout: every
cold start had no primary for ~10 s.
- ha_note_leaderless() marks a config-holding node that becomes a
  follower with no leader known; the liveness poll then retries the
  election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to
  heartbeat_timeout_ms. Primary contact or becoming leader clears it.
- ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity
  then only holds the watchdog's arm time, and the failover gates took
  it for a live primary.
Live cold start: primary 1.7 s after the first node started (was ~10 s).

bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a
standby-only counter a primary never advances, so every primary showed
"appliedLsn: 0". A primary now reports persist->last_applied_lsn. The
controller-status design doc's "already correct in every role" note is
corrected.

Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c
(RED: expected 4900, was 0); controller suite 1256/0. HA soaks:
sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero
job loss).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-28 07:35:45 +00:00
parent 16b422e303
commit aa7fc65a36
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
6 changed files with 260 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

34
src/controller/testsuite/test_ctrlstatus.c
View file

@ -90,6 +90,33 @@ static void cs_teardown_cluster(cluster_t* c)
/* ── tests ──────────────────────────────────────────────────── */
/* bug-1878: a primary never advances ha->applied_lsn (a standby-only
* counter: how far replicated WAL has been received), so ctrlstatus showed
* "committedLsn: 4900 / appliedLsn: 0" on every primary. A primary reports
* its apply cursor, persist->last_applied_lsn (the last entry applied). */
static void test_ctrlstatus_primary_reports_its_apply_cursor(void)
{
cluster_t c;
myconf_t mc;
cs_setup_cluster(&c, &mc);
atomic_store_explicit(&c.ha.role, (int32_t)CTRL_PRIMARY,
memory_order_release);
atomic_store_explicit(&c.ha.applied_lsn, 0, memory_order_release);
atomic_store_explicit(&c.persist.committed_lsn, 4900, memory_order_release);
atomic_store_explicit(&c.persist.last_applied_lsn, 4900,
memory_order_release);
xce_ctrlstatus_t st;
ctrlstatus_fill(&c, &st);
TEST_ASSERT_EQUAL_UINT64(4900, st.committedLsn);
TEST_ASSERT_EQUAL_UINT64(4900, st.appliedLsn);
TEST_ASSERT_TRUE_MESSAGE(st.appliedLsn <= st.committedLsn,
"a primary applies only committed entries");
ctrlstatus_free_fields(&st);
}
/* (a) PRIMARY: responderRole==PRIMARY, primaryName==self, peers/version
* populated; committed/applied/term reflect the atomics. (Req 2.1, 3.1, 3.2,
* 4.1, 4.2, 5.1, 6.1, 6.2, 6.4) */
@ -102,7 +129,9 @@ static void test_ctrlstatus_primary(void)
atomic_store_explicit(&c.ha.role, (int32_t)CTRL_PRIMARY,
memory_order_release);
atomic_store_explicit(&c.ha.term, 7, memory_order_release);
atomic_store_explicit(&c.ha.applied_lsn, 42, memory_order_release);
/* bug-1878: a primary's applied position is persist.last_applied_lsn;
* ha.applied_lsn (standby-only) is left at the 0 a live primary has */
atomic_store_explicit(&c.persist.last_applied_lsn, 41, memory_order_release);
atomic_store_explicit(&c.ha.committed_config_tag, 0xAABBCCDDEE001122ULL,
memory_order_release);
atomic_store_explicit(&c.persist.committed_lsn, 41, memory_order_release);
@ -120,7 +149,7 @@ static void test_ctrlstatus_primary(void)
TEST_ASSERT_NOT_NULL(st.primaryName);
TEST_ASSERT_EQUAL_STRING("ctrlA", st.primaryName);
TEST_ASSERT_EQUAL_UINT64(7, st.term);
TEST_ASSERT_EQUAL_UINT64(42, st.appliedLsn);
TEST_ASSERT_EQUAL_UINT64(41, st.appliedLsn);
TEST_ASSERT_EQUAL_UINT64(41, st.committedLsn);
TEST_ASSERT_EQUAL_UINT64(0xAABBCCDDEE001122ULL, st.configVersion);
TEST_ASSERT_EQUAL_UINT64(0xAABBCCDDEE001122ULL, st.committedConfigVersion);
@ -1098,6 +1127,7 @@ void test_ctrlstatus(void)
UnitySetTestFile(__FILE__);
RUN_TEST(test_ctrlstatus_format_time_ms);
RUN_TEST(test_ctrlstatus_primary);
RUN_TEST(test_ctrlstatus_primary_reports_its_apply_cursor);
RUN_TEST(test_ctrlstatus_standby_known_primary);
RUN_TEST(test_ctrlstatus_unknown_primary);
RUN_TEST(test_ctrlstatus_standby_self_vote_is_unknown);