fix(controller): cold start elects in ~1.7 s, not ~10 s; ctrlstatus appliedLsn on the primary (bug-1879, bug-1878)

bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its
term-1 election before its peers ran; connect failures count as "no"
votes (bug-1423), so it lost at once and became a follower with no
leader. ctrl-2/ctrl-3 hold no config and are voters only (by design),
so nothing retried until ctrl-1's 10 s standby liveness timeout: every
cold start had no primary for ~10 s.
- ha_note_leaderless() marks a config-holding node that becomes a
  follower with no leader known; the liveness poll then retries the
  election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to
  heartbeat_timeout_ms. Primary contact or becoming leader clears it.
- ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity
  then only holds the watchdog's arm time, and the failover gates took
  it for a live primary.
Live cold start: primary 1.7 s after the first node started (was ~10 s).

bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a
standby-only counter a primary never advances, so every primary showed
"appliedLsn: 0". A primary now reports persist->last_applied_lsn. The
controller-status design doc's "already correct in every role" note is
corrected.

Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c
(RED: expected 4900, was 0); controller suite 1256/0. HA soaks:
sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero
job loss).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-28 07:35:45 +00:00
parent 16b422e303
commit aa7fc65a36
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
6 changed files with 260 additions and 6 deletions

View file

@ -175,8 +175,17 @@ void ctrlstatus_fill(cluster_t* c, xce_ctrlstatus_t* st)
atomic_load_explicit(&ha->last_known_committed_config_tag,
memory_order_acquire);
}
st->appliedLsn = atomic_load_explicit(&ha->applied_lsn,
memory_order_acquire);
/* bug-1878: ha->applied_lsn is a standby-only counter (how far
* replicated WAL has been received and persisted); a primary never
* advances it, so every primary reported 0. A primary reports its apply
* cursor instead -- the last entry applied, never above committedLsn. */
if (st->responderRole == (uint8_t)CTRL_PRIMARY) {
st->appliedLsn = atomic_load_explicit(&c->persist.last_applied_lsn,
memory_order_acquire);
} else {
st->appliedLsn = atomic_load_explicit(&ha->applied_lsn,
memory_order_acquire);
}
/* fenced: catch-up OR recovery-unrecoverable (Req 6.3) */
int in_catch_up = atomic_load_explicit(&ha->in_catch_up,