211 lines
14 KiB
Markdown
211 lines
14 KiB
Markdown
# Requirements Document
|
|
|
|
## Project Description (Input)
|
|
|
|
**Who has the problem.** Operators of XCE 3-node controller HA clusters, and the
|
|
controllers themselves, when a controller is temporarily unreachable to its
|
|
peers but not crashed — e.g. a process freeze (SIGSTOP, long GC/allocator
|
|
stall), a one-way or transient network partition, or a slow host — and then
|
|
becomes reachable again ("frozen-then-resumed" / "rejoining" node).
|
|
|
|
**Current situation.** The XCE election layer (Raft library `src/lib/raft/raft.c`
|
|
plus the controller adapter in `src/controller/cluster.c`, `ha.c`, `myapp.c`)
|
|
has no PreVote phase. A node that misses heartbeats hits its liveness timeout
|
|
(`liveness_poll_fn` / RDHUP → `ha_on_primary_ctrl_fd_rdhup`) and immediately
|
|
starts a real election, which **increments its persistent term** before any RPC
|
|
(`raft_start_election`, `raft.c:68`). When such a node rejoins with an inflated
|
|
term and sends `RequestVote`, the live primary adopts the higher term and steps
|
|
down: `raft_on_vote_request` (`raft.c:125-134`, Step 2) updates `term`, sets
|
|
role to follower, and fires `on_become_follower` **before** any freshness or
|
|
role gate. The controller dispatcher (`cluster.c` `OPCode_requestVote`) only
|
|
guards this with the `freshness_ok` check, so a rejoining node that was
|
|
caught-up before the stall is fresh enough to pass and topples the leader.
|
|
Because `raft_cb_become_follower` treats a was-PRIMARY demotion as
|
|
demotion-convergence (truncate-above-committed-LSN + **self-restart**,
|
|
`ha.c`), a single disruptive `RequestVote` makes the healthy primary truncate
|
|
and restart, triggering a multi-term re-election storm.
|
|
|
|
Observed live (2026-06-15, 3-node ASAN deploy): a SIGSTOP'd-then-resumed standby
|
|
forced the primary to demote and self-restart; terms climbed 3 → 6 over ~30s
|
|
before the cluster reconverged. No committed data was lost (`committed_lsn` and
|
|
all submitted jobs survived), and the cluster self-healed — but availability
|
|
blipped for ~30s on an event that should be a no-op. A normal crash+restart does
|
|
NOT trigger this: a restarted node restores its term from disk and observes
|
|
`leader already known → skipping own election` (`myapp.c`). The problem is
|
|
specific to nodes that inflate their term without a genuine leadership change.
|
|
|
|
This is the classic Raft "disruptive server" problem (Raft dissertation §9.6).
|
|
The repository's TLA+ reference (`raft/raft.tla`) and the controller CLAUDE.md
|
|
verification checklist require term monotonicity and Election Safety to be
|
|
preserved by any change to this path.
|
|
|
|
**What should change.** Introduce a PreVote phase so a node cannot inflate its
|
|
term or disrupt a healthy leader merely by becoming temporarily unreachable.
|
|
Before incrementing its term and campaigning, a candidate must first win a
|
|
PreVote majority — a hypothetical "would you vote for me in term+1?" round that
|
|
does NOT change any node's persistent term or recorded vote. Peers grant a
|
|
PreVote only if they (a) would grant the corresponding real vote under the
|
|
existing freshness/log-completeness gate AND (b) have not heard from a current
|
|
leader within the minimum election timeout (leader stickiness). A node that has
|
|
a live primary connection (fresh heartbeat / valid `primary_ctrl_fd`) denies
|
|
PreVotes, so a rejoining node in a cluster with a healthy leader gets no PreVote
|
|
majority, never increments its term, and simply re-attaches as a standby when
|
|
the primary's next heartbeat/ctrlhello arrives.
|
|
|
|
The change must:
|
|
- Preserve all existing Raft safety invariants (Election Safety ≤1 leader/term,
|
|
Leader Completeness, term monotonicity, monotonic commit index) and remain
|
|
consistent with `raft/raft.tla` and the controller verification checklist.
|
|
- Not regress the already-fixed failover (bug-339) and rejoin (bug-340) paths,
|
|
nor the legitimate cases where a real leadership change SHOULD occur (true
|
|
primary death, network healing into a higher-term majority).
|
|
- Integrate with the existing two-tier transport (capnp `RequestVote` /
|
|
`RequestVoteReply` over iomgr, the `ha_elect_ctx_t` vote collection, and the
|
|
raft library callback seams) without breaking the unit-test seams.
|
|
|
|
Scope: the raft library election state machine and its PreVote RPC, the
|
|
controller vote dispatch/freshness/heartbeat-stickiness gates, the election
|
|
trigger path (liveness timeout / RDHUP → PreVote → real election), and the
|
|
wire message (a PreVote flag on the existing vote RPC or a new PreVote message).
|
|
Out of scope: changes to replication, the commit engine, snapshot transfer, or
|
|
config bootstrap beyond what the PreVote gating requires.
|
|
|
|
## Introduction
|
|
|
|
This feature adds a **PreVote phase** to the XCE controller HA election path so
|
|
that a controller which is merely temporarily unreachable cannot inflate its
|
|
term or unseat a healthy primary. Before a node increments its persistent term
|
|
and starts a real election, it must first win a hypothetical PreVote majority.
|
|
Peers grant a PreVote only when the candidate could legitimately become leader
|
|
(the existing freshness / log-completeness gate) **and** the peer no longer
|
|
believes a current leader is alive (leader stickiness). The net effect: a
|
|
frozen-then-resumed standby in a cluster with a healthy primary gets no PreVote
|
|
majority, never increments its term, and re-attaches as a standby — while
|
|
genuine leadership changes (true primary death, healing into a higher-term
|
|
majority) still proceed. The change must preserve every existing Raft safety
|
|
invariant and must not regress the previously fixed failover (bug-339) and
|
|
rejoin (bug-340) behavior.
|
|
|
|
Throughout this document, "the Election Module" refers to the controller's HA
|
|
election state machine (raft library plus its controller adapter) as observed
|
|
through cluster behavior; "the voting controller" refers to a peer evaluating an
|
|
incoming PreVote request; and "the HA cluster" refers to the externally observed
|
|
behavior of the 3-node controller cluster as a whole.
|
|
|
|
## Boundary Context
|
|
|
|
- **In scope**:
|
|
- A PreVote round that precedes any persistent term increment and real
|
|
election on the election-trigger path (liveness timeout / primary-connection
|
|
loss).
|
|
- Voter-side PreVote evaluation combining the existing freshness /
|
|
log-completeness gate with a new leader-stickiness gate.
|
|
- The user-/operator-observable outcomes for frozen-then-resumed rejoin,
|
|
genuine failover, and partition healing.
|
|
- PreVote requests and replies traveling over the existing election transport.
|
|
- **Out of scope**:
|
|
- Replication, the commit/barrier engine, snapshot/WAL catch-up transfer, and
|
|
config bootstrap — except where PreVote gating must read state those paths
|
|
already expose.
|
|
- Redesign of the existing freshness / log-completeness comparison itself; this
|
|
feature reuses it as the real-vote eligibility criterion.
|
|
- **Adjacent expectations** (relied on, not owned by this feature):
|
|
- The existing freshness / log-completeness gate (config-tag and last-log-LSN
|
|
floor) remains the authority on whether a candidate could win a real vote.
|
|
- The existing heartbeat-liveness / primary-connection tracking remains the
|
|
signal for whether a peer still believes a leader is alive.
|
|
- The existing two-tier election transport and vote-collection mechanism
|
|
continue to deliver and tally PreVote and real-vote messages.
|
|
- Term restore-from-disk on crash+restart continues to suppress unnecessary
|
|
elections; this feature does not change that path.
|
|
|
|
## Requirements
|
|
|
|
### Requirement 1: PreVote precedes any persistent term increment
|
|
|
|
**Objective:** As an XCE controller node, I want to win a hypothetical PreVote
|
|
majority before starting a real election, so that I never inflate my persistent
|
|
term unless a genuine leadership change is actually possible.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When a node's election trigger fires (liveness/heartbeat timeout or primary-connection RDHUP), the Election Module shall initiate a PreVote round before incrementing its persistent term.
|
|
2. While a PreVote round is in progress, the Election Module shall not modify the node's persistent term or recorded vote.
|
|
3. When a node requests PreVotes, the Election Module shall request them for the prospective next term (current term + 1) without persisting that prospective term.
|
|
4. If a node does not obtain a PreVote majority, then the Election Module shall not start a real election and shall not increment its persistent term.
|
|
5. When a node obtains a PreVote majority, the Election Module shall start a real election by incrementing its persistent term and casting its self-vote, following the existing real-election path.
|
|
|
|
### Requirement 2: Voter-side PreVote eligibility reuses the real-vote gate
|
|
|
|
**Objective:** As a voting controller, I want to grant a PreVote only to a
|
|
candidate I would actually elect, so that PreVote screens out candidates that
|
|
could never legitimately become leader.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When a voting controller receives a PreVote request, the voting controller shall grant it only if the candidate would also pass the existing real-vote freshness and log-completeness gate (config-tag and last-log-LSN floor).
|
|
2. If a PreVote candidate fails the freshness or log-completeness gate, then the voting controller shall deny the PreVote.
|
|
3. When a voting controller grants or denies a PreVote, the voting controller shall not record a vote, shall not change its voted-for state, and shall not change its persistent term.
|
|
4. When a voting controller has granted a PreVote to one candidate, the voting controller shall remain able to grant a real vote later in the same term to any eligible candidate, regardless of which candidate received the PreVote.
|
|
|
|
### Requirement 3: Leader stickiness gate
|
|
|
|
**Objective:** As a voting controller that still believes its leader is alive, I
|
|
want to deny PreVotes, so that a temporarily-unreachable peer cannot disrupt a
|
|
healthy primary.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. While a voting controller still considers a current leader alive — it has received leader traffic recently enough that it would not itself start an election — the voting controller shall deny all PreVote requests, even when the candidate passes the freshness gate.
|
|
2. If a voting controller has gone long enough without leader traffic that it would itself be eligible to start an election, then the voting controller shall evaluate PreVote requests under the freshness / log-completeness gate alone.
|
|
3. While a controller is itself the current primary holding quorum, the controller shall deny all PreVote requests.
|
|
|
|
### Requirement 4: Frozen-then-resumed rejoin is non-disruptive
|
|
|
|
**Objective:** As an operator, I want a frozen-then-resumed standby to rejoin
|
|
without disrupting the cluster, so that transient unreachability is effectively a
|
|
no-op for availability.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When a previously-frozen standby resumes and initiates a PreVote round while a healthy primary still holds the cluster, the HA cluster shall deny it a PreVote majority so that it neither increments its term nor sends a real RequestVote.
|
|
2. When a non-disruptive rejoining standby next receives a primary heartbeat or ctrlhello, the standby shall re-attach as a standby under the existing primary without any intervening persistent-term change.
|
|
3. While a healthy primary holds quorum, the HA cluster shall not demote, truncate-and-converge, or self-restart the primary in response to a rejoining standby's election attempt.
|
|
4. When a frozen-then-resumed event occurs in an otherwise healthy 3-node cluster, the HA cluster shall keep the cluster's persistent term unchanged across the event and shall not enter a multi-term re-election cycle.
|
|
|
|
### Requirement 5: Legitimate leadership changes still proceed
|
|
|
|
**Objective:** As an XCE HA cluster, I want genuine leadership changes to
|
|
complete, so that PreVote suppresses only disruptive elections and never blocks
|
|
real failover.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When the primary genuinely fails and no peer has heard from a leader within its election-eligibility window, the HA cluster shall grant PreVotes to an eligible candidate so that a real election proceeds and a new primary is elected.
|
|
2. When a partition heals and a node observes a higher-term majority, the Election Module shall still adopt the higher term and step down according to the existing term-monotonicity rules.
|
|
3. The Election Module shall preserve the previously fixed failover behavior (bug-339) and rejoin behavior (bug-340) with PreVote enabled.
|
|
4. Where a real leadership change is warranted, the HA cluster shall complete the new election within the existing election-timeout bounds, introducing no additional unbounded delay from the PreVote round.
|
|
|
|
### Requirement 6: Raft safety invariants preserved
|
|
|
|
**Objective:** As an XCE HA cluster, I want every existing Raft safety invariant
|
|
preserved, so that adding PreVote introduces no correctness regression.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. The Election Module shall preserve Election Safety (at most one leader per term) across all PreVote and real-vote paths.
|
|
2. The Election Module shall preserve term monotonicity, such that no PreVote causes any node's persistent term to decrease and no node's persistent term increases without a completed real election.
|
|
3. The HA cluster shall preserve Leader Completeness and a monotonic commit index across PreVote-gated elections.
|
|
4. The PreVote behavior shall remain consistent with the repository's TLA+ reference (`raft/raft.tla`) and the controller verification checklist for term monotonicity and Election Safety.
|
|
|
|
### Requirement 7: PreVote transport compatibility and operator observability
|
|
|
|
**Objective:** As an operator, I want PreVote activity to be diagnosable and to
|
|
not break existing election exchanges, so that I can distinguish a suppressed
|
|
disruptive election from a real failover.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. The HA cluster shall carry PreVote requests and replies over the existing election transport without disrupting real RequestVote / RequestVoteReply exchanges.
|
|
2. When a controller denies a PreVote due to leader stickiness or the freshness gate, the HA cluster shall record the denial with the candidate identity and the denial reason at a diagnosable log level.
|
|
3. When a node abandons a campaign after failing to obtain a PreVote majority, the HA cluster shall record that outcome so operators can distinguish a suppressed disruptive election from a genuine failover.
|