bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its
term-1 election before its peers ran; connect failures count as "no"
votes (bug-1423), so it lost at once and became a follower with no
leader. ctrl-2/ctrl-3 hold no config and are voters only (by design),
so nothing retried until ctrl-1's 10 s standby liveness timeout: every
cold start had no primary for ~10 s.
- ha_note_leaderless() marks a config-holding node that becomes a
follower with no leader known; the liveness poll then retries the
election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to
heartbeat_timeout_ms. Primary contact or becoming leader clears it.
- ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity
then only holds the watchdog's arm time, and the failover gates took
it for a live primary.
Live cold start: primary 1.7 s after the first node started (was ~10 s).
bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a
standby-only counter a primary never advances, so every primary showed
"appliedLsn: 0". A primary now reports persist->last_applied_lsn. The
controller-status design doc's "already correct in every role" note is
corrected.
Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c
(RED: expected 4900, was 0); controller suite 1256/0. HA soaks:
sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero
job loss).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Found live during task 8.5's optional catching-up-observation attempt:
a killed-then-reconnected standby can end up occupying a NEW
replicator_t.standbys[] slot while its OLD slot (host/port still
populated, ep == NULL) also matches the same configured peer.
ctrlstatus_fill's primary-side peer-status loop stopped at the first
host:port match, so it permanently reported that peer dead even
minutes after it had genuinely reconnected and fully caught up --
violating Requirement 13.3's "true, currently-known connection status"
guarantee.
Fix: scan all MAX_STANDBYS slots for a given peer and prefer any slot
with ep != NULL over a slot with ep == NULL; fall back to dead only
when no matching slot is live. Read-only, lock scope unchanged from
task 8.2.
TDD RED->GREEN (1216 Tests: 2 Failures -> 0), independently reproduced
by the reviewer (exact RED replay) plus two independent live
kill/backlog/restart/poll cycles on separate fresh clusters, both
confirming the previously-stuck standby now converges to ok.
design.md's Locked Decision 3 corrected to document the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Populates CtrlPeer.status (added in task 8.1) inside ctrlstatus_fill's
peer-building loop: self always reports ok; a primary responder sources
every other peer from the live replication connection registry
(replicator_t.standbys[], host+port matched, reusing
repl_active_standby_count's ep!=NULL && !in_catch_up predicate) --
dead/catching-up/ok; a non-primary responder reports unknown for every
non-self peer. A self-identity mismatch (pre-existing misconfiguration)
logs one WARN and falls through to the normal non-self sourcing rather
than forcing any peer to ok. Read-only: a bounded lock-protected scan,
no state mutation.
TDD: 3 new tests (primary-sourced-from-registry, standby-unknown-except-
self, self-mismatch-falls-through) plus an extended no-mutation
assertion; RED->GREEN independently reproduced by the reviewer.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
ctrlstatus_fill now unconditionally reads ha->last_known_elected_at_ms
into the reply's elected_at_ms field -- one atomic-acquire load, no
role-branch, mirroring how the handler already reports raw term
unconditionally. Serves both a self-stamped primary and a
heartbeat-learned standby from the same field. "Unknown" display
gating is deliberately left to the client command (task 7.5), not
this handler.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Mirror the sibling last_known_committed_lsn: all writes release, reads acquire.
Prereq for a heartbeat-receive writer + the ctrlstatus reader to interact
safely (Req 2.3). Pure access migration, no behavior change; controller suite
0 failures.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Role-branch the committed sourcing: PRIMARY reports its own authoritative
persist.committed_lsn / committed_config_tag; off-primary reports the LEARNED
ha->last_known_committed_lsn / last_known_committed_config_tag instead of the
primary-only raw markers that read 0 on a standby (fixes the applied>committed
inversion). Stamp responder_time_ms from CLOCK_REALTIME for log correlation.
Read-only (Req 10.1-10.4, 11.1). 4 new TDD tests; controller suite 110/0/0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
