# Requirements Document ## Project Description (Input) **Who has the problem.** Operators of XCE 3-node controller HA clusters, and the controllers themselves, when a controller is temporarily unreachable to its peers but not crashed — e.g. a process freeze (SIGSTOP, long GC/allocator stall), a one-way or transient network partition, or a slow host — and then becomes reachable again ("frozen-then-resumed" / "rejoining" node). **Current situation.** The XCE election layer (Raft library `src/lib/raft/raft.c` plus the controller adapter in `src/controller/cluster.c`, `ha.c`, `myapp.c`) has no PreVote phase. A node that misses heartbeats hits its liveness timeout (`liveness_poll_fn` / RDHUP → `ha_on_primary_ctrl_fd_rdhup`) and immediately starts a real election, which **increments its persistent term** before any RPC (`raft_start_election`, `raft.c:68`). When such a node rejoins with an inflated term and sends `RequestVote`, the live primary adopts the higher term and steps down: `raft_on_vote_request` (`raft.c:125-134`, Step 2) updates `term`, sets role to follower, and fires `on_become_follower` **before** any freshness or role gate. The controller dispatcher (`cluster.c` `OPCode_requestVote`) only guards this with the `freshness_ok` check, so a rejoining node that was caught-up before the stall is fresh enough to pass and topples the leader. Because `raft_cb_become_follower` treats a was-PRIMARY demotion as demotion-convergence (truncate-above-committed-LSN + **self-restart**, `ha.c`), a single disruptive `RequestVote` makes the healthy primary truncate and restart, triggering a multi-term re-election storm. Observed live (2026-06-15, 3-node ASAN deploy): a SIGSTOP'd-then-resumed standby forced the primary to demote and self-restart; terms climbed 3 → 6 over ~30s before the cluster reconverged. No committed data was lost (`committed_lsn` and all submitted jobs survived), and the cluster self-healed — but availability blipped for ~30s on an event that should be a no-op. A normal crash+restart does NOT trigger this: a restarted node restores its term from disk and observes `leader already known → skipping own election` (`myapp.c`). The problem is specific to nodes that inflate their term without a genuine leadership change. This is the classic Raft "disruptive server" problem (Raft dissertation §9.6). The repository's TLA+ reference (`raft/raft.tla`) and the controller CLAUDE.md verification checklist require term monotonicity and Election Safety to be preserved by any change to this path. **What should change.** Introduce a PreVote phase so a node cannot inflate its term or disrupt a healthy leader merely by becoming temporarily unreachable. Before incrementing its term and campaigning, a candidate must first win a PreVote majority — a hypothetical "would you vote for me in term+1?" round that does NOT change any node's persistent term or recorded vote. Peers grant a PreVote only if they (a) would grant the corresponding real vote under the existing freshness/log-completeness gate AND (b) have not heard from a current leader within the minimum election timeout (leader stickiness). A node that has a live primary connection (fresh heartbeat / valid `primary_ctrl_fd`) denies PreVotes, so a rejoining node in a cluster with a healthy leader gets no PreVote majority, never increments its term, and simply re-attaches as a standby when the primary's next heartbeat/ctrlhello arrives. The change must: - Preserve all existing Raft safety invariants (Election Safety ≤1 leader/term, Leader Completeness, term monotonicity, monotonic commit index) and remain consistent with `raft/raft.tla` and the controller verification checklist. - Not regress the already-fixed failover (bug-339) and rejoin (bug-340) paths, nor the legitimate cases where a real leadership change SHOULD occur (true primary death, network healing into a higher-term majority). - Integrate with the existing two-tier transport (capnp `RequestVote` / `RequestVoteReply` over iomgr, the `ha_elect_ctx_t` vote collection, and the raft library callback seams) without breaking the unit-test seams. Scope: the raft library election state machine and its PreVote RPC, the controller vote dispatch/freshness/heartbeat-stickiness gates, the election trigger path (liveness timeout / RDHUP → PreVote → real election), and the wire message (a PreVote flag on the existing vote RPC or a new PreVote message). Out of scope: changes to replication, the commit engine, snapshot transfer, or config bootstrap beyond what the PreVote gating requires. ## Introduction This feature adds a **PreVote phase** to the XCE controller HA election path so that a controller which is merely temporarily unreachable cannot inflate its term or unseat a healthy primary. Before a node increments its persistent term and starts a real election, it must first win a hypothetical PreVote majority. Peers grant a PreVote only when the candidate could legitimately become leader (the existing freshness / log-completeness gate) **and** the peer no longer believes a current leader is alive (leader stickiness). The net effect: a frozen-then-resumed standby in a cluster with a healthy primary gets no PreVote majority, never increments its term, and re-attaches as a standby — while genuine leadership changes (true primary death, healing into a higher-term majority) still proceed. The change must preserve every existing Raft safety invariant and must not regress the previously fixed failover (bug-339) and rejoin (bug-340) behavior. Throughout this document, "the Election Module" refers to the controller's HA election state machine (raft library plus its controller adapter) as observed through cluster behavior; "the voting controller" refers to a peer evaluating an incoming PreVote request; and "the HA cluster" refers to the externally observed behavior of the 3-node controller cluster as a whole. ## Boundary Context - **In scope**: - A PreVote round that precedes any persistent term increment and real election on the election-trigger path (liveness timeout / primary-connection loss). - Voter-side PreVote evaluation combining the existing freshness / log-completeness gate with a new leader-stickiness gate. - The user-/operator-observable outcomes for frozen-then-resumed rejoin, genuine failover, and partition healing. - PreVote requests and replies traveling over the existing election transport. - **Out of scope**: - Replication, the commit/barrier engine, snapshot/WAL catch-up transfer, and config bootstrap — except where PreVote gating must read state those paths already expose. - Redesign of the existing freshness / log-completeness comparison itself; this feature reuses it as the real-vote eligibility criterion. - **Adjacent expectations** (relied on, not owned by this feature): - The existing freshness / log-completeness gate (config-tag and last-log-LSN floor) remains the authority on whether a candidate could win a real vote. - The existing heartbeat-liveness / primary-connection tracking remains the signal for whether a peer still believes a leader is alive. - The existing two-tier election transport and vote-collection mechanism continue to deliver and tally PreVote and real-vote messages. - Term restore-from-disk on crash+restart continues to suppress unnecessary elections; this feature does not change that path. ## Requirements ### Requirement 1: PreVote precedes any persistent term increment **Objective:** As an XCE controller node, I want to win a hypothetical PreVote majority before starting a real election, so that I never inflate my persistent term unless a genuine leadership change is actually possible. #### Acceptance Criteria 1. When a node's election trigger fires (liveness/heartbeat timeout or primary-connection RDHUP), the Election Module shall initiate a PreVote round before incrementing its persistent term. 2. While a PreVote round is in progress, the Election Module shall not modify the node's persistent term or recorded vote. 3. When a node requests PreVotes, the Election Module shall request them for the prospective next term (current term + 1) without persisting that prospective term. 4. If a node does not obtain a PreVote majority, then the Election Module shall not start a real election and shall not increment its persistent term. 5. When a node obtains a PreVote majority, the Election Module shall start a real election by incrementing its persistent term and casting its self-vote, following the existing real-election path. ### Requirement 2: Voter-side PreVote eligibility reuses the real-vote gate **Objective:** As a voting controller, I want to grant a PreVote only to a candidate I would actually elect, so that PreVote screens out candidates that could never legitimately become leader. #### Acceptance Criteria 1. When a voting controller receives a PreVote request, the voting controller shall grant it only if the candidate would also pass the existing real-vote freshness and log-completeness gate (config-tag and last-log-LSN floor). 2. If a PreVote candidate fails the freshness or log-completeness gate, then the voting controller shall deny the PreVote. 3. When a voting controller grants or denies a PreVote, the voting controller shall not record a vote, shall not change its voted-for state, and shall not change its persistent term. 4. When a voting controller has granted a PreVote to one candidate, the voting controller shall remain able to grant a real vote later in the same term to any eligible candidate, regardless of which candidate received the PreVote. ### Requirement 3: Leader stickiness gate **Objective:** As a voting controller that still believes its leader is alive, I want to deny PreVotes, so that a temporarily-unreachable peer cannot disrupt a healthy primary. #### Acceptance Criteria 1. While a voting controller still considers a current leader alive — it has received leader traffic recently enough that it would not itself start an election — the voting controller shall deny all PreVote requests, even when the candidate passes the freshness gate. 2. If a voting controller has gone long enough without leader traffic that it would itself be eligible to start an election, then the voting controller shall evaluate PreVote requests under the freshness / log-completeness gate alone. 3. While a controller is itself the current primary holding quorum, the controller shall deny all PreVote requests. ### Requirement 4: Frozen-then-resumed rejoin is non-disruptive **Objective:** As an operator, I want a frozen-then-resumed standby to rejoin without disrupting the cluster, so that transient unreachability is effectively a no-op for availability. #### Acceptance Criteria 1. When a previously-frozen standby resumes and initiates a PreVote round while a healthy primary still holds the cluster, the HA cluster shall deny it a PreVote majority so that it neither increments its term nor sends a real RequestVote. 2. When a non-disruptive rejoining standby next receives a primary heartbeat or ctrlhello, the standby shall re-attach as a standby under the existing primary without any intervening persistent-term change. 3. While a healthy primary holds quorum, the HA cluster shall not demote, truncate-and-converge, or self-restart the primary in response to a rejoining standby's election attempt. 4. When a frozen-then-resumed event occurs in an otherwise healthy 3-node cluster, the HA cluster shall keep the cluster's persistent term unchanged across the event and shall not enter a multi-term re-election cycle. ### Requirement 5: Legitimate leadership changes still proceed **Objective:** As an XCE HA cluster, I want genuine leadership changes to complete, so that PreVote suppresses only disruptive elections and never blocks real failover. #### Acceptance Criteria 1. When the primary genuinely fails and no peer has heard from a leader within its election-eligibility window, the HA cluster shall grant PreVotes to an eligible candidate so that a real election proceeds and a new primary is elected. 2. When a partition heals and a node observes a higher-term majority, the Election Module shall still adopt the higher term and step down according to the existing term-monotonicity rules. 3. The Election Module shall preserve the previously fixed failover behavior (bug-339) and rejoin behavior (bug-340) with PreVote enabled. 4. Where a real leadership change is warranted, the HA cluster shall complete the new election within the existing election-timeout bounds, introducing no additional unbounded delay from the PreVote round. ### Requirement 6: Raft safety invariants preserved **Objective:** As an XCE HA cluster, I want every existing Raft safety invariant preserved, so that adding PreVote introduces no correctness regression. #### Acceptance Criteria 1. The Election Module shall preserve Election Safety (at most one leader per term) across all PreVote and real-vote paths. 2. The Election Module shall preserve term monotonicity, such that no PreVote causes any node's persistent term to decrease and no node's persistent term increases without a completed real election. 3. The HA cluster shall preserve Leader Completeness and a monotonic commit index across PreVote-gated elections. 4. The PreVote behavior shall remain consistent with the repository's TLA+ reference (`raft/raft.tla`) and the controller verification checklist for term monotonicity and Election Safety. ### Requirement 7: PreVote transport compatibility and operator observability **Objective:** As an operator, I want PreVote activity to be diagnosable and to not break existing election exchanges, so that I can distinguish a suppressed disruptive election from a real failover. #### Acceptance Criteria 1. The HA cluster shall carry PreVote requests and replies over the existing election transport without disrupting real RequestVote / RequestVoteReply exchanges. 2. When a controller denies a PreVote due to leader stickiness or the freshness gate, the HA cluster shall record the denial with the candidate identity and the denial reason at a diagnosable log level. 3. When a node abandons a campaign after failing to obtain a PreVote majority, the HA cluster shall record that outcome so operators can distinguish a suppressed disruptive election from a genuine failover.