lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): cold start elects in ~1.7 s, not ~10 s; ctrlstatus appliedLsn on the primary (bug-1879, bug-1878)

Browse source 
bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its
term-1 election before its peers ran; connect failures count as "no"
votes (bug-1423), so it lost at once and became a follower with no
leader. ctrl-2/ctrl-3 hold no config and are voters only (by design),
so nothing retried until ctrl-1's 10 s standby liveness timeout: every
cold start had no primary for ~10 s.
- ha_note_leaderless() marks a config-holding node that becomes a
  follower with no leader known; the liveness poll then retries the
  election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to
  heartbeat_timeout_ms. Primary contact or becoming leader clears it.
- ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity
  then only holds the watchdog's arm time, and the failover gates took
  it for a live primary.
Live cold start: primary 1.7 s after the first node started (was ~10 s).

bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a
standby-only counter a primary never advances, so every primary showed
"appliedLsn: 0". A primary now reports persist->last_applied_lsn. The
controller-status design doc's "already correct in every role" note is
corrected.

Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c
(RED: expected 4900, was 0); controller suite 1256/0. HA soaks:
sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero
job loss).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-28 07:35:45 +00:00
parent 16b422e303
commit aa7fc65a36
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
6 changed files with 260 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

2
.kiro/specs/controller-status/design.md
View file

@ -273,7 +273,7 @@ No change to `committedLsn @8` / `committedConfigVersion @7` wire shape — Req
- **Req 10 (committed sourcing):** replace the unconditional reads at `ctrlstatus.c:108–110,130` with the role-branch above:
- `responderRole == CTRL_PRIMARY` → `committedLsn = persist.committed_lsn`, `committedConfigVersion = ha->committed_config_tag` (atomic acquire loads, unchanged).
- else → `committedLsn = ha->last_known_committed_lsn` (atomic acquire), `committedConfigVersion = ha->last_known_committed_config_tag` (plain aligned load).
- `appliedLsn` (`ha->applied_lsn`) is unchanged — it is already correct in every role.
- `appliedLsn` (`ha->applied_lsn`) is unchanged here. **Correction (bug-1878, 2026-09-28):** it is *not* correct in every role — `ha->applied_lsn` is a standby-only counter (replicated WAL received/persisted) that a primary never advances, so every primary reported `appliedLsn: 0`. `ctrlstatus_fill` now reports a primary's apply cursor, `persist->last_applied_lsn` (the last entry applied, never above `committedLsn`); a standby still reports `ha->applied_lsn`.
- **Req 11 (timestamp):** stamp `st->responder_time_ms` from `clock_gettime(CLOCK_REALTIME)` (`tv_sec*1000 + tv_nsec/1e6`) at the top of the snapshot, before/with the other field loads, so it marks the responder's sample moment.
- **Read-only invariant (10.4, 1.2) preserved:** all additions are atomic/aligned loads + a clock read; no HA/persist/config mutation.