bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its
term-1 election before its peers ran; connect failures count as "no"
votes (bug-1423), so it lost at once and became a follower with no
leader. ctrl-2/ctrl-3 hold no config and are voters only (by design),
so nothing retried until ctrl-1's 10 s standby liveness timeout: every
cold start had no primary for ~10 s.
- ha_note_leaderless() marks a config-holding node that becomes a
follower with no leader known; the liveness poll then retries the
election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to
heartbeat_timeout_ms. Primary contact or becoming leader clears it.
- ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity
then only holds the watchdog's arm time, and the failover gates took
it for a live primary.
Live cold start: primary 1.7 s after the first node started (was ~10 s).
bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a
standby-only counter a primary never advances, so every primary showed
"appliedLsn: 0". A primary now reports persist->last_applied_lsn. The
controller-status design doc's "already correct in every role" note is
corrected.
Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c
(RED: expected 4900, was 0); controller suite 1256/0. HA soaks:
sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero
job loss).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Found live during task 8.5's optional catching-up-observation attempt:
a killed-then-reconnected standby can end up occupying a NEW
replicator_t.standbys[] slot while its OLD slot (host/port still
populated, ep == NULL) also matches the same configured peer.
ctrlstatus_fill's primary-side peer-status loop stopped at the first
host:port match, so it permanently reported that peer dead even
minutes after it had genuinely reconnected and fully caught up --
violating Requirement 13.3's "true, currently-known connection status"
guarantee.
Fix: scan all MAX_STANDBYS slots for a given peer and prefer any slot
with ep != NULL over a slot with ep == NULL; fall back to dead only
when no matching slot is live. Read-only, lock scope unchanged from
task 8.2.
TDD RED->GREEN (1216 Tests: 2 Failures -> 0), independently reproduced
by the reviewer (exact RED replay) plus two independent live
kill/backlog/restart/poll cycles on separate fresh clusters, both
confirming the previously-stuck standby now converges to ok.
design.md's Locked Decision 3 corrected to document the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Second design-review pass caught a factual error: the Constraints
section claimed ctrlstatus_fill "already takes ha->lock" for the
primary-identity read, citing the ORIGINAL Req 1-9 design text rather
than the as-built code -- after the ade0081 fix, that read takes no
lock at all. Corrected: this delta's replicator_t.lock acquisition is
the first lock ctrlstatus_fill has taken since that fix, not a repeat
of an existing in-function pattern (still safe, per the existing
repl_active_standby_count precedent elsewhere in the codebase).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds requirements, design, and research for showing each configured
controller's connection status (ok/catching-up/dead/unknown) in
ctrlstatus's peer list, with the same off-primary fidelity pattern
already established (primary reports the true picture; a standby
reports unknown for what it cannot itself observe).
Design review found and fixed one gap before implementation: no
defined fallback if the responder's own name doesn't match any
configured peer entry (a pre-existing misconfiguration class, not new
to this delta) -- now an explicit, logged (WARN) degradation instead
of a silently wrong status.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Feature-level /kiro-validate-impl found the delta's File Structure Plan
omitted 5 as-built files task 7.3 actually needed (request.h/request.c
-- the real heartbeat_msg_t struct-builder -- plus 3 cascading
signature-conformance test files). Documents the as-built reality and
the already-fixed cmd_ctrlstatus gate text; also notes the pre-existing,
deliberate lack of a checked-in unit test for cmd_ctrlstatus's print
layer (the class of gap bug-1818 exploited) as a known, accepted scope
boundary for future readers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Live 3-node E2E validation (task 7.6) found that cmd_ctrlstatus's
electedTerm/electedAt lines (task 7.5) gated on NULLSTR(primaryName),
which is unconditionally empty on every non-primary responder in this
build (ctrlstatus_fill only self-reports the primary's own name --
ha->primary_host is vestigial, a pre-existing Req 2 behavior). This made
electedTerm/electedAt print "unknown" on every standby, forever, even
though elected_at_ms was correctly, non-zero, and convergently populated
the whole time (term/committedLsn/appliedLsn, sourced the same
unconditional way, DID correctly display and converge on the same query).
Fix: gate on status->elected_at_ms == 0 instead -- the field stays
exactly 0 until a real promotion stamps it (either road) or a heartbeat
carries a learned value from an actual primary, unlike term (which can
legitimately bump from mere vote/candidacy activity before any primary
is elected). ctrlstatus_fill itself is untouched; this is purely a
cmd_ctrlstatus display-layer correction.
Also corrects design.md's "Design Delta -- Requirement 12" Locked
Decision 3 (and its dependent traceability/service-contract text) to
document this live-E2E finding, mirroring the spec's existing delta
convention (Req 10/11's own live-E2E correction).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds requirements, design, research, and tasks for reporting the term
and wall-clock time the current primary was elected, including
standby-side learned fidelity (mirroring the Req 10 pattern), following
the existing controller-status spec's delta convention (Req 10/11).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
