fix(ha): election-safety hardening — vote gates, restored-vote term, freshness source
Election protocol fixes validated against raft.tla invariants:
- ha_init_raft(): one-time raft state init at startup (re-init would lose
durable election progress); n_peers = full cluster size including self
- handle_request_vote(): higher-term step-down moved BEFORE the role gate
(split-brain fix); standby vote-deny now requires a live primary control
channel (primary_ctrl_fd >= 0) AND a fresh heartbeat
- ha_heartbeat_is_fresh(): derive freshness from ha->last_activity (the
field production actually writes); the orphaned last_heartbeat timespec
had no production writer, leaving the vote-deny gate inert (bug-337);
field removed from ha_state_t
- handle_request_vote(): restore same-candidate idempotent re-grant —
deny only a DIFFERENT candidate in the same term, matching raft.c and
Raft §5.2 votedFor ∈ {Nil, candidateId} (bug-338)
- raft_init(): restore voted_for_term alongside a persisted vote; leaving
it 0 allowed a double vote in the same term across a restart (bug-339)
- myapp_run(): leader_known check treats a vote recorded under this node's
own name as a self-vote — production persists the real candidate name,
the "self" placeholder only appears in test mode
- raft_on_vote_request(): leader steps down on higher-term vote request
instead of denying it
Tests: test_proto_H6_restored_vote_blocks_second_vote added; H1 restore
assertion strengthened; test_ha_vote_gate.c driven through last_activity +
primary_ctrl_fd (the real production fields). Full /xce-build all green
(lib + controller + scheduler + Go suites, 0 failures, ASAN clean).
deploy-ha.sh: start.sh now forwards extra args to the controller binary.
This commit is contained in:
parent
fa66d1ba9c
commit
9821ec4f3c
7 changed files with 207 additions and 91 deletions
|
|
@ -217,6 +217,11 @@ int myapp_run(myconf_t *myconf)
|
|||
uint64_t my_lsn = (uint64_t)atomic_load_explicit(&c.ha.applied_lsn,
|
||||
memory_order_acquire);
|
||||
|
||||
/* Initialize raft state machine once with cluster topology.
|
||||
* Must happen before any election or watchdog path that calls
|
||||
* raft_on_leader_lost() / raft_start_election(). */
|
||||
ha_init_raft(&c.ha, &c, n_peers);
|
||||
|
||||
/* Startup stagger: delay election start by priority * election_priority_step_ms.
|
||||
* Node 0 starts first; lower-priority nodes wait, see the winner's vote request
|
||||
* arrive, and grant the vote instead of starting their own election.
|
||||
|
|
@ -240,11 +245,24 @@ int myapp_run(myconf_t *myconf)
|
|||
* path (iomgr was already running); the raft term + primary_ctrl_fd checks
|
||||
* catch the case where iomgr was NOT running during the nap (connections
|
||||
* queued in the kernel backlog) but the winner's ctrlhello has already
|
||||
* established the control channel. */
|
||||
* established the control channel.
|
||||
*
|
||||
* voted_for_name is combined with term: a self-vote means this node was
|
||||
* a candidate, not a follower — no external leader is known. In
|
||||
* production raft_cb_persist records the self-vote under this node's own
|
||||
* name (myconf->name); the "self" placeholder only appears in test mode,
|
||||
* so both spellings must be checked. A stale vote (voted_for_term <
|
||||
* ha->term) is also discarded. All these cases must fall through to
|
||||
* ha_start_election_async. */
|
||||
{
|
||||
int leader_known = 0;
|
||||
pthread_mutex_lock(&c.ha.lock);
|
||||
if (c.ha.voted_for_name[0] != '\0')
|
||||
uint64_t vft = c.ha.voted_for_term;
|
||||
uint64_t cur = atomic_load_explicit(&c.ha.term, memory_order_relaxed);
|
||||
if (c.ha.voted_for_name[0] != '\0' &&
|
||||
strcmp(c.ha.voted_for_name, "self") != 0 &&
|
||||
strcmp(c.ha.voted_for_name, myconf->name) != 0 &&
|
||||
vft == cur)
|
||||
leader_known = 1;
|
||||
pthread_mutex_unlock(&c.ha.lock);
|
||||
if (!leader_known && c.ha.raft.term > 0)
|
||||
|
|
@ -253,10 +271,10 @@ int myapp_run(myconf_t *myconf)
|
|||
leader_known = 1;
|
||||
if (leader_known) {
|
||||
INFO("myapp_run: leader already known after startup delay "
|
||||
"(voted_for=%s, raft_term=%" PRIu64 ", primary_fd=%d) — "
|
||||
"skipping own election",
|
||||
"(voted_for=%s, voted_term=%" PRIu64 ", raft_term=%" PRIu64
|
||||
", primary_fd=%d) — skipping own election",
|
||||
c.ha.voted_for_name[0] ? c.ha.voted_for_name : "(none)",
|
||||
c.ha.raft.term, c.ha.primary_ctrl_fd);
|
||||
vft, c.ha.raft.term, c.ha.primary_ctrl_fd);
|
||||
goto skip_election;
|
||||
}
|
||||
}
|
||||
|
|
@ -281,6 +299,11 @@ skip_election:;
|
|||
atomic_store_explicit(&c.ha.role, (int32_t)CTRL_STANDBY,
|
||||
memory_order_release);
|
||||
INFO("myapp_run: CTRL_STANDBY — waiting for primary to connect and push config");
|
||||
/* Arm heartbeat watchdog even when we skipped the election.
|
||||
* If the known primary never connects within heartbeat_timeout_ms,
|
||||
* the watchdog fires ha_on_primary_ctrl_fd_rdhup which increments
|
||||
* the term (clearing voted_for) and starts a fresh election. */
|
||||
ha_arm_heartbeat_timeout_vtask(&c.ha, &c);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue