fix(ha): election-safety hardening — vote gates, restored-vote term, freshness source

Election protocol fixes validated against raft.tla invariants:

- ha_init_raft(): one-time raft state init at startup (re-init would lose
  durable election progress); n_peers = full cluster size including self
- handle_request_vote(): higher-term step-down moved BEFORE the role gate
  (split-brain fix); standby vote-deny now requires a live primary control
  channel (primary_ctrl_fd >= 0) AND a fresh heartbeat
- ha_heartbeat_is_fresh(): derive freshness from ha->last_activity (the
  field production actually writes); the orphaned last_heartbeat timespec
  had no production writer, leaving the vote-deny gate inert (bug-337);
  field removed from ha_state_t
- handle_request_vote(): restore same-candidate idempotent re-grant —
  deny only a DIFFERENT candidate in the same term, matching raft.c and
  Raft §5.2 votedFor ∈ {Nil, candidateId} (bug-338)
- raft_init(): restore voted_for_term alongside a persisted vote; leaving
  it 0 allowed a double vote in the same term across a restart (bug-339)
- myapp_run(): leader_known check treats a vote recorded under this node's
  own name as a self-vote — production persists the real candidate name,
  the "self" placeholder only appears in test mode
- raft_on_vote_request(): leader steps down on higher-term vote request
  instead of denying it

Tests: test_proto_H6_restored_vote_blocks_second_vote added; H1 restore
assertion strengthened; test_ha_vote_gate.c driven through last_activity +
primary_ctrl_fd (the real production fields). Full /xce-build all green
(lib + controller + scheduler + Go suites, 0 failures, ASAN clean).

deploy-ha.sh: start.sh now forwards extra args to the controller binary.
This commit is contained in:
Rongsong Shen 2026-06-10 01:25:39 +00:00
parent fa66d1ba9c
commit 9821ec4f3c
7 changed files with 207 additions and 91 deletions

View file

@ -217,6 +217,11 @@ int myapp_run(myconf_t *myconf)
uint64_t my_lsn = (uint64_t)atomic_load_explicit(&c.ha.applied_lsn,
memory_order_acquire);
/* Initialize raft state machine once with cluster topology.
* Must happen before any election or watchdog path that calls
* raft_on_leader_lost() / raft_start_election(). */
ha_init_raft(&c.ha, &c, n_peers);
/* Startup stagger: delay election start by priority * election_priority_step_ms.
* Node 0 starts first; lower-priority nodes wait, see the winner's vote request
* arrive, and grant the vote instead of starting their own election.
@ -240,11 +245,24 @@ int myapp_run(myconf_t *myconf)
* path (iomgr was already running); the raft term + primary_ctrl_fd checks
* catch the case where iomgr was NOT running during the nap (connections
* queued in the kernel backlog) but the winner's ctrlhello has already
* established the control channel. */
* established the control channel.
*
* voted_for_name is combined with term: a self-vote means this node was
* a candidate, not a follower — no external leader is known. In
* production raft_cb_persist records the self-vote under this node's own
* name (myconf->name); the "self" placeholder only appears in test mode,
* so both spellings must be checked. A stale vote (voted_for_term <
* ha->term) is also discarded. All these cases must fall through to
* ha_start_election_async. */
{
int leader_known = 0;
pthread_mutex_lock(&c.ha.lock);
if (c.ha.voted_for_name[0] != '\0')
uint64_t vft = c.ha.voted_for_term;
uint64_t cur = atomic_load_explicit(&c.ha.term, memory_order_relaxed);
if (c.ha.voted_for_name[0] != '\0' &&
strcmp(c.ha.voted_for_name, "self") != 0 &&
strcmp(c.ha.voted_for_name, myconf->name) != 0 &&
vft == cur)
leader_known = 1;
pthread_mutex_unlock(&c.ha.lock);
if (!leader_known && c.ha.raft.term > 0)
@ -253,10 +271,10 @@ int myapp_run(myconf_t *myconf)
leader_known = 1;
if (leader_known) {
INFO("myapp_run: leader already known after startup delay "
"(voted_for=%s, raft_term=%" PRIu64 ", primary_fd=%d) — "
"skipping own election",
"(voted_for=%s, voted_term=%" PRIu64 ", raft_term=%" PRIu64
", primary_fd=%d) — skipping own election",
c.ha.voted_for_name[0] ? c.ha.voted_for_name : "(none)",
c.ha.raft.term, c.ha.primary_ctrl_fd);
vft, c.ha.raft.term, c.ha.primary_ctrl_fd);
goto skip_election;
}
}
@ -281,6 +299,11 @@ skip_election:;
atomic_store_explicit(&c.ha.role, (int32_t)CTRL_STANDBY,
memory_order_release);
INFO("myapp_run: CTRL_STANDBY — waiting for primary to connect and push config");
/* Arm heartbeat watchdog even when we skipped the election.
* If the known primary never connects within heartbeat_timeout_ms,
* the watchdog fires ha_on_primary_ctrl_fd_rdhup which increments
* the term (clearing voted_for) and starts a fresh election. */
ha_arm_heartbeat_timeout_vtask(&c.ha, &c);
}
}