myxce/.kiro/specs/ha-prevote/requirements.md

Requirements Document

Project Description (Input)

Who has the problem. Operators of XCE 3-node controller HA clusters, and the controllers themselves, when a controller is temporarily unreachable to its peers but not crashed — e.g. a process freeze (SIGSTOP, long GC/allocator stall), a one-way or transient network partition, or a slow host — and then becomes reachable again ("frozen-then-resumed" / "rejoining" node).

Current situation. The XCE election layer (Raft library src/lib/raft/raft.c plus the controller adapter in src/controller/cluster.c, ha.c, myapp.c) has no PreVote phase. A node that misses heartbeats hits its liveness timeout (liveness_poll_fn / RDHUP → ha_on_primary_ctrl_fd_rdhup) and immediately starts a real election, which increments its persistent term before any RPC (raft_start_election, raft.c:68). When such a node rejoins with an inflated term and sends RequestVote, the live primary adopts the higher term and steps down: raft_on_vote_request (raft.c:125-134, Step 2) updates term, sets role to follower, and fires on_become_follower before any freshness or role gate. The controller dispatcher (cluster.c OPCode_requestVote) only guards this with the freshness_ok check, so a rejoining node that was caught-up before the stall is fresh enough to pass and topples the leader. Because raft_cb_become_follower treats a was-PRIMARY demotion as demotion-convergence (truncate-above-committed-LSN + self-restart, ha.c), a single disruptive RequestVote makes the healthy primary truncate and restart, triggering a multi-term re-election storm.

Observed live (2026-06-15, 3-node ASAN deploy): a SIGSTOP'd-then-resumed standby forced the primary to demote and self-restart; terms climbed 3 → 6 over ~30s before the cluster reconverged. No committed data was lost (committed_lsn and all submitted jobs survived), and the cluster self-healed — but availability blipped for ~30s on an event that should be a no-op. A normal crash+restart does NOT trigger this: a restarted node restores its term from disk and observes leader already known → skipping own election (myapp.c). The problem is specific to nodes that inflate their term without a genuine leadership change.

This is the classic Raft "disruptive server" problem (Raft dissertation §9.6). The repository's TLA+ reference (raft/raft.tla) and the controller CLAUDE.md verification checklist require term monotonicity and Election Safety to be preserved by any change to this path.

What should change. Introduce a PreVote phase so a node cannot inflate its term or disrupt a healthy leader merely by becoming temporarily unreachable. Before incrementing its term and campaigning, a candidate must first win a PreVote majority — a hypothetical "would you vote for me in term+1?" round that does NOT change any node's persistent term or recorded vote. Peers grant a PreVote only if they (a) would grant the corresponding real vote under the existing freshness/log-completeness gate AND (b) have not heard from a current leader within the minimum election timeout (leader stickiness). A node that has a live primary connection (fresh heartbeat / valid primary_ctrl_fd) denies PreVotes, so a rejoining node in a cluster with a healthy leader gets no PreVote majority, never increments its term, and simply re-attaches as a standby when the primary's next heartbeat/ctrlhello arrives.

The change must:

• Preserve all existing Raft safety invariants (Election Safety ≤1 leader/term, Leader Completeness, term monotonicity, monotonic commit index) and remain consistent with raft/raft.tla and the controller verification checklist.
• Not regress the already-fixed failover (bug-339) and rejoin (bug-340) paths, nor the legitimate cases where a real leadership change SHOULD occur (true primary death, network healing into a higher-term majority).
• Integrate with the existing two-tier transport (capnp RequestVote / RequestVoteReply over iomgr, the ha_elect_ctx_t vote collection, and the raft library callback seams) without breaking the unit-test seams.

Scope: the raft library election state machine and its PreVote RPC, the controller vote dispatch/freshness/heartbeat-stickiness gates, the election trigger path (liveness timeout / RDHUP → PreVote → real election), and the wire message (a PreVote flag on the existing vote RPC or a new PreVote message). Out of scope: changes to replication, the commit engine, snapshot transfer, or config bootstrap beyond what the PreVote gating requires.

Introduction

This feature adds a PreVote phase to the XCE controller HA election path so that a controller which is merely temporarily unreachable cannot inflate its term or unseat a healthy primary. Before a node increments its persistent term and starts a real election, it must first win a hypothetical PreVote majority. Peers grant a PreVote only when the candidate could legitimately become leader (the existing freshness / log-completeness gate) and the peer no longer believes a current leader is alive (leader stickiness). The net effect: a frozen-then-resumed standby in a cluster with a healthy primary gets no PreVote majority, never increments its term, and re-attaches as a standby — while genuine leadership changes (true primary death, healing into a higher-term majority) still proceed. The change must preserve every existing Raft safety invariant and must not regress the previously fixed failover (bug-339) and rejoin (bug-340) behavior.

Throughout this document, "the Election Module" refers to the controller's HA election state machine (raft library plus its controller adapter) as observed through cluster behavior; "the voting controller" refers to a peer evaluating an incoming PreVote request; and "the HA cluster" refers to the externally observed behavior of the 3-node controller cluster as a whole.

Boundary Context

• In scope:
  • A PreVote round that precedes any persistent term increment and real election on the election-trigger path (liveness timeout / primary-connection loss).
  • Voter-side PreVote evaluation combining the existing freshness / log-completeness gate with a new leader-stickiness gate.
  • The user-/operator-observable outcomes for frozen-then-resumed rejoin, genuine failover, and partition healing.
  • PreVote requests and replies traveling over the existing election transport.
• Out of scope:
  • Replication, the commit/barrier engine, snapshot/WAL catch-up transfer, and config bootstrap — except where PreVote gating must read state those paths already expose.
  • Redesign of the existing freshness / log-completeness comparison itself; this feature reuses it as the real-vote eligibility criterion.
• Adjacent expectations (relied on, not owned by this feature):
  • The existing freshness / log-completeness gate (config-tag and last-log-LSN floor) remains the authority on whether a candidate could win a real vote.
  • The existing heartbeat-liveness / primary-connection tracking remains the signal for whether a peer still believes a leader is alive.
  • The existing two-tier election transport and vote-collection mechanism continue to deliver and tally PreVote and real-vote messages.
  • Term restore-from-disk on crash+restart continues to suppress unnecessary elections; this feature does not change that path.

Requirements

Requirement 1: PreVote precedes any persistent term increment

Objective: As an XCE controller node, I want to win a hypothetical PreVote majority before starting a real election, so that I never inflate my persistent term unless a genuine leadership change is actually possible.

Acceptance Criteria

1. When a node's election trigger fires (liveness/heartbeat timeout or primary-connection RDHUP), the Election Module shall initiate a PreVote round before incrementing its persistent term.
2. While a PreVote round is in progress, the Election Module shall not modify the node's persistent term or recorded vote.
3. When a node requests PreVotes, the Election Module shall request them for the prospective next term (current term + 1) without persisting that prospective term.
4. If a node does not obtain a PreVote majority, then the Election Module shall not start a real election and shall not increment its persistent term.
5. When a node obtains a PreVote majority, the Election Module shall start a real election by incrementing its persistent term and casting its self-vote, following the existing real-election path.

Requirement 2: Voter-side PreVote eligibility reuses the real-vote gate

Objective: As a voting controller, I want to grant a PreVote only to a candidate I would actually elect, so that PreVote screens out candidates that could never legitimately become leader.

Acceptance Criteria

1. When a voting controller receives a PreVote request, the voting controller shall grant it only if the candidate would also pass the existing real-vote freshness and log-completeness gate (config-tag and last-log-LSN floor).
2. If a PreVote candidate fails the freshness or log-completeness gate, then the voting controller shall deny the PreVote.
3. When a voting controller grants or denies a PreVote, the voting controller shall not record a vote, shall not change its voted-for state, and shall not change its persistent term.
4. When a voting controller has granted a PreVote to one candidate, the voting controller shall remain able to grant a real vote later in the same term to any eligible candidate, regardless of which candidate received the PreVote.

Requirement 3: Leader stickiness gate

Objective: As a voting controller that still believes its leader is alive, I want to deny PreVotes, so that a temporarily-unreachable peer cannot disrupt a healthy primary.

Acceptance Criteria

1. While a voting controller still considers a current leader alive — it has received leader traffic recently enough that it would not itself start an election — the voting controller shall deny all PreVote requests, even when the candidate passes the freshness gate.
2. If a voting controller has gone long enough without leader traffic that it would itself be eligible to start an election, then the voting controller shall evaluate PreVote requests under the freshness / log-completeness gate alone.
3. While a controller is itself the current primary holding quorum, the controller shall deny all PreVote requests.

Requirement 4: Frozen-then-resumed rejoin is non-disruptive

Objective: As an operator, I want a frozen-then-resumed standby to rejoin without disrupting the cluster, so that transient unreachability is effectively a no-op for availability.

Acceptance Criteria

1. When a previously-frozen standby resumes and initiates a PreVote round while a healthy primary still holds the cluster, the HA cluster shall deny it a PreVote majority so that it neither increments its term nor sends a real RequestVote.
2. When a non-disruptive rejoining standby next receives a primary heartbeat or ctrlhello, the standby shall re-attach as a standby under the existing primary without any intervening persistent-term change.
3. While a healthy primary holds quorum, the HA cluster shall not demote, truncate-and-converge, or self-restart the primary in response to a rejoining standby's election attempt.
4. When a frozen-then-resumed event occurs in an otherwise healthy 3-node cluster, the HA cluster shall keep the cluster's persistent term unchanged across the event and shall not enter a multi-term re-election cycle.

Requirement 5: Legitimate leadership changes still proceed

Objective: As an XCE HA cluster, I want genuine leadership changes to complete, so that PreVote suppresses only disruptive elections and never blocks real failover.

Acceptance Criteria

1. When the primary genuinely fails and no peer has heard from a leader within its election-eligibility window, the HA cluster shall grant PreVotes to an eligible candidate so that a real election proceeds and a new primary is elected.
2. When a partition heals and a node observes a higher-term majority, the Election Module shall still adopt the higher term and step down according to the existing term-monotonicity rules.
3. The Election Module shall preserve the previously fixed failover behavior (bug-339) and rejoin behavior (bug-340) with PreVote enabled.
4. Where a real leadership change is warranted, the HA cluster shall complete the new election within the existing election-timeout bounds, introducing no additional unbounded delay from the PreVote round.

Requirement 6: Raft safety invariants preserved

Objective: As an XCE HA cluster, I want every existing Raft safety invariant preserved, so that adding PreVote introduces no correctness regression.

Acceptance Criteria

1. The Election Module shall preserve Election Safety (at most one leader per term) across all PreVote and real-vote paths.
2. The Election Module shall preserve term monotonicity, such that no PreVote causes any node's persistent term to decrease and no node's persistent term increases without a completed real election.
3. The HA cluster shall preserve Leader Completeness and a monotonic commit index across PreVote-gated elections.
4. The PreVote behavior shall remain consistent with the repository's TLA+ reference (raft/raft.tla) and the controller verification checklist for term monotonicity and Election Safety.

Requirement 7: PreVote transport compatibility and operator observability

Objective: As an operator, I want PreVote activity to be diagnosable and to not break existing election exchanges, so that I can distinguish a suppressed disruptive election from a real failover.

Acceptance Criteria

1. The HA cluster shall carry PreVote requests and replies over the existing election transport without disrupting real RequestVote / RequestVoteReply exchanges.
2. When a controller denies a PreVote due to leader stickiness or the freshness gate, the HA cluster shall record the denial with the candidate identity and the denial reason at a diagnosable log level.
3. When a node abandons a campaign after failing to obtain a PreVote majority, the HA cluster shall record that outcome so operators can distinguish a suppressed disruptive election from a genuine failover.