myxce/.kiro/specs/ha-prevote/design.md

Design Document

Overview

Purpose. This feature adds a PreVote phase to the XCE controller HA election path so that a controller which is merely temporarily unreachable cannot inflate its persistent term or unseat a healthy primary. Before a node increments its term and campaigns, it must first win a hypothetical PreVote majority — a "would you vote for me in term+1?" round that mutates no node's persistent term or recorded vote.

Users. Operators of 3-node controller HA clusters, and the controllers themselves, benefit: a frozen-then-resumed standby rejoins as a no-op instead of triggering the ~30s demote-and-restart re-election storm observed live on 2026-06-15.

Impact. The election trigger path changes from trigger → real election (term++) to trigger → PreVote round → (only on majority) → real election (term++). The real-vote state machine, replication, and commit engine are unchanged. The change is additive: two new wire opcodes reusing the existing vote payload structs, pure PreVote decision and tally functions in the raft library (mirroring the production real-vote path raft_on_vote_reply), and PreVote orchestration in the controller that reuses the existing eligibility gate.

Goals

• A node never increments its persistent term unless a PreVote majority is first obtained.
• A voter denies a PreVote whenever it still believes a leader is alive (leader stickiness), reusing the existing heartbeat-freshness predicate.
• A frozen-then-resumed standby in a healthy cluster gets no PreVote majority and re-attaches as a standby with the cluster term unchanged.
• Genuine leadership changes (true primary death, partition heal) still elect within existing timeout bounds.
• All Raft safety invariants and the bug-339/bug-340 fixes are preserved.

Non-Goals

• Changes to replication, the commit/barrier engine, snapshot/WAL catch-up transfer, or config bootstrap.
• Redesign of the freshness / log-completeness comparison (reused as-is).
• Backward/rolling-upgrade compatibility (product is pre-release; all controllers run the same build).
• Leader-side CheckQuorum self-demotion and the defense-in-depth real-vote stickiness guard (see Architecture → Deferred decisions).
• The startup / first-boot election (the ha_elect_ctx_t.done semaphore path). PreVote gates only the failover trigger; on first boot there is no leader to protect and every node's heartbeat is stale, so a PreVote round would trivially pass. The startup election path is untouched.

Boundary Commitments

This Spec Owns

• The PreVote round on the election-trigger path: a side-effect-free majority probe that runs before any persistent term increment.
• The voter-side PreVote decision: term check + reuse of the existing eligibility gate + reuse of the dispatcher freshness gate, with zero persistent mutation.
• The two new wire opcodes (preVoteRequest, preVoteReply) and their dispatch.
• The trigger-path restructuring so that truncate-WAL + CTRL_ELECTING happens only after PreVote succeeds.
• PreVote observability (denial reasons, abandoned-campaign logging).

Out of Boundary

• The real-vote state machine (raft_on_vote_request, raft_on_vote_reply, raft_start_election) — invoked unchanged after PreVote success.
• The freshness floor computation (ha_vote_self_freshness) and the eligibility gate (raft_cb_vote_eligible) — reused, not modified (except that the eligibility gate is now also called from the PreVote receiver path).
• Replication, commit barrier, snapshot transfer, demotion-convergence mechanics (truncate-above-committed + self-restart) — unchanged.
• Leader self-demotion / CheckQuorum.
• The startup-election ha_elect_ctx_t.done semaphore path — not gated by PreVote.

Allowed Dependencies

• raft library callback seams (raft_callbacks_t), the raft-library vote-tally pattern (raft_on_vote_reply ⇒ raft_on_pre_vote_reply), ha_heartbeat_is_fresh, ha_vote_self_freshness, the capnp election transport (xce_request / vote endpoints / peer_ctrl_msg_handler), and the xce_vote_request_hook test seam.
• Dependency direction (must not be violated): capnp/types → raft library (pure) → controller adapter (ha.c) → dispatch/transport (cluster.c). The raft library must not depend on controller state.

Revalidation Triggers

• Any change to the request_vote_req_t / request_vote_reply_t payload shape (shared by real-vote and PreVote).
• Any change to raft_cb_vote_eligible semantics (now shared by both vote paths).
• Any change to the leader-stickiness predicate (ha_heartbeat_is_fresh / heartbeat_timeout_ms).
• Any change to the election-trigger entry points (ha_on_primary_ctrl_fd_rdhup, liveness_poll_fn).
• Introduction of mixed-version clusters (would reintroduce the deferred wire-compat concern).

Architecture

Existing Architecture Analysis

The election path is split across two layers (preserved by this design):

• raft library (src/lib/raft/raft.c, raft.h) — the canonical, TLA+-checkable state machine. Owns term, voted_for, role, and the pure vote-decision logic. Communicates outward only through raft_callbacks_t (no controller dependency). Today: raft_start_election increments+persists the term before any RPC; raft_on_vote_request adopts a higher term and steps down a leader in Step 2 — before the on_vote_eligible gate runs. This ordering is exactly why the existing heartbeat-stickiness gate cannot protect a leader from a fresh, higher-term RequestVote, and why the fix must be sender-side.
• controller adapter (src/controller/ha.c, cluster.c) — async transport + collection. cluster.c dispatches OPCode_requestVote, applies the freshness floor, maps candidate name→index, then calls the library. ha.c orchestrates ha_start_election_async (allocates ha_elect_ctx_t, self-vote, delegates to raft_start_election), collects replies (handle_request_vote_reply → ha_elect_evaluate_and_signal), and gates eligibility (raft_cb_vote_eligible: PRIMARY-deny, fresh-heartbeat-deny, catch-up-deny). Triggers: ha_on_primary_ctrl_fd_rdhup (RDHUP) and liveness_poll_fn (heartbeat timeout), which today transition to CTRL_ELECTING, truncate WAL, and start the election.

Architecture Pattern & Boundary Map

graph TB
    subgraph trigger [Election Trigger]
        Rdhup[ha_on_primary_ctrl_fd_rdhup]
        Liveness[liveness_poll_fn]
    end
    subgraph controller [Controller Adapter ha.c cluster.c]
        BeginFailover[ha_begin_failover]
        PrevoteAsync[ha_start_prevote_async]
        ElectAsync[ha_start_election_async]
        Collect[raft_on_pre_vote_reply tally]
        DispReq[cluster.c preVoteRequest case]
        DispRep[cluster.c preVoteReply case]
        Eligible[raft_cb_vote_eligible reused]
        Fresh[ha_vote_self_freshness reused]
    end
    subgraph raftlib [Raft Library pure]
        Begin[raft_begin_pre_vote]
        OnReq[raft_on_pre_vote_request]
        OnRep[raft_on_pre_vote_reply]
        StartElection[raft_start_election unchanged]
    end
    subgraph wire [Transport]
        Capnp[preVoteRequest preVoteReply opcodes]
    end

    Rdhup --> BeginFailover
    Liveness --> BeginFailover
    BeginFailover --> PrevoteAsync
    PrevoteAsync --> Begin
    Begin --> Capnp
    Capnp --> DispReq
    DispReq --> Fresh
    DispReq --> Eligible
    DispReq --> OnReq
    OnReq --> Capnp
    Capnp --> DispRep
    DispRep --> OnRep
    OnRep --> Collect
    Collect -->|majority| ElectAsync
    ElectAsync --> StartElection
    Collect -->|lost or timeout| Rearm[re-arm liveness stay STANDBY]

Architecture Integration

• Selected pattern: side-effect-free probe interposed before an existing state transition (Raft dissertation §9.6 PreVote). Adopted, not invented.
• Boundaries: raft library owns the pure PreVote decision and tally; the controller owns the orchestration/transport. The voter eligibility gate is shared with the real-vote path via reuse, not duplication; the PreVote tally mirrors the production real-vote tally (raft_on_vote_reply) rather than the test-only ha_elect_ctx_t path.
• Existing patterns preserved: callback-seam isolation of the raft library; raft-library-owns-tally (raft_on_vote_reply ⇒ raft_on_pre_vote_reply); dispatcher-applies-freshness-before-library; xce_vote_request_hook test seam.
• New components rationale: a new opcode pair keeps the real-vote handler byte-for-byte unchanged (zero regression risk to bug-339/340); pure PreVote functions in the library keep the state machine TLA+-checkable.
• Steering compliance: async, non-blocking vtask handlers; long-lived connections; inbound connections vote-only; immutable-term discipline (PreVote mutates nothing).

Deferred decisions (documented, not implemented)

• Defense-in-depth real-vote stickiness guard — rejecting a real higher-term RequestVote while a leader is believed alive. PreVote already guarantees a candidate only sends a real RequestVote after a majority's stickiness has lapsed, so in a same-version cluster no stickiness-violating real RequestVote can arise. Adding the guard would touch the safety-critical real-vote term-adoption path and risk regressing bug-339/340 for no in-scope benefit. Deferred; revisit only if mixed-version clusters or leader CheckQuorum are introduced.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Messaging / Events	Cap'n Proto (capnpc-c 1.0.1)	New preVoteRequest/preVoteReply opcodes reusing existing vote payload structs	No new structs; follows existing opcode-enum pattern
Backend / Services	C (raft library + controller)	Pure PreVote decision + tally functions; controller orchestration	No new libraries; reuses iomgr/delaymgr and the raft tally pattern
Infrastructure / Runtime	iomgr async transport, delaymgr	PreVote send/collect + round timeout	Non-blocking, mirrors election path

File Structure Plan

Modified Files

• src/schema/controller.capnp — add two OPCode enum entries: preVoteRequest, preVoteReply. Reuse existing RequestVoteRequest / RequestVoteReply structs and their Request.body union arms (requestvotemsg / requestvotereplymsg); no new payload structs.
• src/lib/raft/raft.h — add pre-vote return codes (e.g. RAFT_PRE_VOTE_WON, RAFT_PRE_VOTE_LOST, RAFT_PRE_VOTE_PENDING), an optional on_send_pre_vote_request callback in raft_callbacks_t, pre-vote tally fields in raft_state_t (pre_votes_granted, pre_vote_replies, pre_vote_in_progress), and declarations for the three new functions.
• src/lib/raft/raft.c — add raft_begin_pre_vote, raft_on_pre_vote_request, raft_on_pre_vote_reply (all pure: no on_persist, no term/voted_for/role mutation). raft_on_pre_vote_request reuses on_vote_eligible. raft_on_pre_vote_reply owns the PreVote tally in raft_state_t (mirroring raft_on_vote_reply) and returns WON/LOST/PENDING — it never fires on_become_leader/on_become_follower.
• src/controller/ha.h — add a controller-side PreVote round handle on ha_state_t (round-timeout context + in-progress flag for cancellation). No change to ha_elect_ctx_t (the tally lives in raft_state_t, not here).
• src/controller/ha.c — add ha_begin_failover (common entry), ha_start_prevote_async (calls raft_begin_pre_vote, arms round timeout), and pre_vote_reply_dispatch (calls raft_on_pre_vote_reply, mirroring vote_reply_dispatch → raft_on_vote_reply); on RAFT_PRE_VOTE_WON perform truncate + CTRL_ELECTING + ha_start_election_async; on RAFT_PRE_VOTE_LOST/round-timeout stay CTRL_STANDBY, re-arm liveness, log abandonment. Modify ha_on_primary_ctrl_fd_rdhup and liveness_poll_fn to call ha_begin_failover (PreVote first; no truncate/CTRL_ELECTING until PreVote wins); modify ha_on_heartbeat_received to cancel an in-progress PreVote and re-attach.
• src/controller/cluster.c — add OPCode_preVoteRequest and OPCode_preVoteReply dispatch cases mirroring the real-vote cases, calling raft_on_pre_vote_request (no term adoption) and routing replies to pre_vote_reply_dispatch.

New Files

• src/controller/testsuite/test_ha_prevote.c — PreVote gating + frozen-then-resumed no-op scenario + genuine-failover scenario.

Test files extended

• src/lib/testsuite/test_raft.c — pure PreVote decision/tally unit tests + invariant assertions (no term mutation).
• src/controller/testsuite/test_ha_vote_gate.c — PreVote eligibility reuse of the role/heartbeat/catch-up gates.

System Flows

PreVote round (candidate side)

sequenceDiagram
    participant Trig as Trigger
    participant Cand as Candidate ha.c
    participant Peer as Peer voter
    Trig->>Cand: ha_begin_failover (stay CTRL_STANDBY)
    Cand->>Cand: ha_start_prevote_async (prospective term = current+1, no persist)
    Cand->>Peer: preVoteRequest term+1 tag lsn
    Note over Peer: stickiness gate then freshness gate then term check
    Peer-->>Cand: preVoteReply granted or denied (peer term unchanged)
    alt PreVote majority
        Cand->>Cand: now truncate WAL, CTRL_ELECTING
        Cand->>Cand: ha_start_election_async then raft_start_election (term++)
    else lost or round timeout
        Cand->>Cand: stay CTRL_STANDBY, re-arm liveness watchdog
        Note over Cand: next primary heartbeat re-attaches as standby
    end

Key decisions: the candidate remains CTRL_STANDBY and mutates nothing during the PreVote round; the destructive truncate + CTRL_ELECTING step is deferred to PreVote success. An incoming primary heartbeat during the round cancels it and re-attaches. The round is bounded by election_timeout_ms; on loss the watchdog is re-armed so a later genuine failure still triggers, and repeated PreVote attempts succeed once a majority's stickiness lapses.

PreVote decision (voter side)

flowchart TD
    A[preVoteRequest received] --> B{role == CTRL_PRIMARY}
    B -- yes --> D[deny]
    B -- no --> C{role == CTRL_STANDBY and heartbeat fresh}
    C -- yes --> D
    C -- no --> E{in catch-up}
    E -- yes --> D
    E -- no --> F{req.term lt current_term}
    F -- yes --> D
    F -- no --> G{freshness floor ok}
    G -- no --> D
    G -- yes --> H[grant, no term change, no persist, no voted_for]

Steps B/C/E are exactly the existing raft_cb_vote_eligible gates (reused). Step G is exactly the existing dispatcher freshness floor (ha_vote_self_freshness). The only PreVote-specific rule is the absence of any state mutation on either grant or deny.

Requirements Traceability

Requirement	Summary	Components	Interfaces / Flows
1.1	PreVote before term increment on trigger	ha_begin_failover, ha_start_prevote_async	PreVote round flow
1.2	No term/vote mutation during round	raft_begin_pre_vote, ha_start_prevote_async	candidate stays STANDBY
1.3	Request prospective term+1, not persisted	raft_begin_pre_vote	preVoteRequest payload
1.4	No majority → no real election	raft_on_pre_vote_reply (LOST), pre_vote_reply_dispatch	round flow "lost" path
1.5	Majority → start real election	ha_start_election_async, raft_start_election (unchanged)	round flow "won" path
2.1	Grant only if real-vote freshness gate passes	cluster.c preVoteRequest, ha_vote_self_freshness	voter flow step G
2.2	Fail freshness → deny	same	voter flow step G
2.3	Grant/deny records no vote, no term	raft_on_pre_vote_request	voter flow leaf H
2.4	PreVote does not consume the real vote	raft_on_pre_vote_request (no voted_for write)	—
3.1	Deny while leader believed alive	raft_cb_vote_eligible (ha_heartbeat_is_fresh)	voter flow step C
3.2	Past self-election threshold → freshness-only eval	raft_cb_vote_eligible	voter flow C→E→F→G
3.3	Primary denies PreVotes	raft_cb_vote_eligible (PRIMARY-deny)	voter flow step B
4.1	Frozen-resumed gets no majority	full path	frozen-rejoin scenario test
4.2	Re-attach on next heartbeat, no term change	ha_on_heartbeat_received	round flow "lost"/cancel
4.3	Healthy primary not demoted/restarted	PreVote denies → no real RequestVote	—
4.4	Term unchanged, no re-election cycle	candidate stays STANDBY	—
5.1	True death → PreVotes granted → elect	raft_cb_vote_eligible (stale heartbeat → eligible)	genuine-failover test
5.2	Partition heal → adopt higher term	raft_on_vote_reply (unchanged)	real-vote path
5.3	No regression of bug-339/bug-340	real-vote path untouched	—
5.4	Complete within election-timeout bounds	election_timeout_ms round timeout	round flow
6.1	Election Safety preserved	real-vote path untouched; PreVote grants no leadership	RN-1 (research.md)
6.2	Term monotonicity; PreVote never changes term	raft_on_pre_vote_* (pure)	—
6.3	Leader Completeness, monotonic commit index	freshness floor reused; commit path untouched	—
6.4	Consistent with raft.tla + checklist	test_raft.c invariant tests	RN-1
7.1	PreVote over existing transport, no disruption to real votes	new opcodes, reused arms	—
7.2	Log PreVote denials with candidate + reason	cluster.c preVoteRequest, on_trace/tlog	—
7.3	Log abandoned campaign	pre_vote_reply_dispatch (LOST/timeout)	round flow

Components and Interfaces

Component	Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
raft pre-vote functions	raft library	Pure PreVote decision + tally	1.2, 1.3, 2.3, 2.4, 6.2	on_vote_eligible (P0), on_send_pre_vote_request (P1)	Service, State
PreVote dispatch	cluster.c	Receive/answer PreVote, no mutation	2.1, 2.2, 3.1–3.3, 7.1, 7.2	ha_vote_self_freshness (P0), raft_on_pre_vote_request (P0)	Service, Event
PreVote orchestration	ha.c	Run round, gate trigger, branch to election	1.1, 1.4, 1.5, 4.x, 5.4, 7.3	raft_on_pre_vote_reply (P0), ha_start_election_async (P0), liveness (P1)	Service, State
Trigger restructuring	ha.c	Defer truncate/CTRL_ELECTING until PreVote wins	1.1, 4.2, 4.3, 4.4	ha_on_primary_ctrl_fd_rdhup, liveness_poll_fn (P0)	State

Raft library — pre-vote functions

Field	Detail
Intent	Pure PreVote decision and tally; canonical, TLA+-checkable; mutates no persistent state
Requirements	1.2, 1.3, 2.3, 2.4, 6.2

Responsibilities & Constraints

• raft_begin_pre_vote: compute prospective term = term + 1 without assigning it to s->term or persisting; set pre_vote_in_progress, pre_votes_granted = 1 (self); emit on_send_pre_vote_request(ctx, peer_idx, prospective_term) per peer.
• raft_on_pre_vote_request: deny if cand_term < s->term; else return the result of on_vote_eligible(ctx, from_idx, cand_term); never touches term, voted_for, role, or calls on_persist. Freshness is applied by the dispatcher before this call (same contract as real vote).
• raft_on_pre_vote_reply: owns the PreVote tally in raft_state_t (pre_votes_granted, pre_vote_replies), mirroring raft_on_vote_reply; return RAFT_PRE_VOTE_WON at quorum, RAFT_PRE_VOTE_LOST when all replied without quorum, else RAFT_PRE_VOTE_PENDING. Unlike raft_on_vote_reply, it never fires on_become_leader/on_become_follower and makes no role/term change — the controller acts on the returned status.
• Invariant: across all three functions, s->term, s->voted_for, s->voted_for_term, and s->role are unchanged, and on_persist is never invoked.

Contracts: Service [x] / State [x]

Service Interface

/* All pure: no persist, no term/role/voted_for mutation. */
int raft_begin_pre_vote(raft_state_t *s);                 /* sends pre-votes, self-grant in raft_state_t */
int raft_on_pre_vote_request(raft_state_t *s,
                             int from_idx, uint64_t cand_term);  /* 1=grant, 0=deny */
int raft_on_pre_vote_reply(raft_state_t *s,
                           int peer_idx, int granted);    /* tally in raft_state_t -> WON / LOST / PENDING */

• Preconditions: raft_begin_pre_vote requires role == RAFT_FOLLOWER.
• Postconditions: on RAFT_PRE_VOTE_WON the caller invokes the unchanged raft_start_election.
• Invariants: persistent state untouched (asserted in test_raft.c).

Implementation Notes

• Integration: on_vote_eligible is the existing raft_cb_vote_eligible; reusing it gives PRIMARY-deny + heartbeat-stickiness + catch-up-deny for free (Req 3.x).
• Validation: unit tests assert term/voted_for unchanged after a granted PreVote.
• Risks: accidental term mutation — mitigated by keeping all term writes in raft_start_election only.

Controller — PreVote dispatch (cluster.c)

Field	Detail
Intent	Answer preVoteRequest with freshness + stickiness gates, no mutation; route preVoteReply to the tally
Requirements	2.1, 2.2, 3.1–3.3, 7.1, 7.2

Responsibilities & Constraints

• OPCode_preVoteRequest: parse request_vote_req_t; compute freshness floor via ha_vote_self_freshness; deny on freshness failure; otherwise call raft_on_pre_vote_request; reply request_vote_reply_t with unchanged term, the grant bit, and peer_config_tag/peer_last_log_lsn. Log denials with candidate id + reason (Req 7.2).
• OPCode_preVoteReply: route to pre_vote_reply_dispatch → raft_on_pre_vote_reply (raft-library tally), paralleling the production vote_reply_dispatch → raft_on_vote_reply.
• Inbound connections remain vote-only; the reply carries no state push.

Contracts: Service [x] / Event [x]

Event Contract

• Published: preVoteRequest (term+1, candidate_id, config_tag, last_log_lsn).
• Subscribed: preVoteReply (term, vote_granted, peer_config_tag, peer_last_log_lsn).
• Delivery: best-effort over the existing election transport; missing/denied replies count as non-grants. No ordering requirement beyond per-peer idempotency: raft_on_pre_vote_reply receives peer_idx and tracks a per-peer replied flag in raft_state_t to avoid double-counting duplicate replies.

Controller — PreVote orchestration & trigger restructuring (ha.c)

Field	Detail
Intent	Run the side-effect-free round, gate the failover trigger, collect replies, branch to real election or clean abandon
Requirements	1.1, 1.4, 1.5, 4.1–4.4, 5.4, 7.3

Responsibilities & Constraints

• ha_begin_failover: single entry from both ha_on_primary_ctrl_fd_rdhup and liveness_poll_fn; starts ha_start_prevote_async; does not truncate WAL or transition to CTRL_ELECTING (deferred to PreVote success).
• ha_start_prevote_async: call raft_begin_pre_vote (self-grant + send preVoteRequest per peer via on_send_pre_vote_request); arm an election_timeout_ms round timeout. The PreVote tally lives in raft_state_t, not a controller struct.
• pre_vote_reply_dispatch → raft_on_pre_vote_reply: on RAFT_PRE_VOTE_WON → truncate + CTRL_ELECTING + ha_start_election_async; on RAFT_PRE_VOTE_LOST or round timeout → stay CTRL_STANDBY, re-arm liveness, log abandonment (Req 7.3).
• ha_on_heartbeat_received: if a PreVote round is in progress, cancel it (clear the round handle) and re-attach as standby (Req 4.2) — no term change.
• Invariant: while a PreVote round is in progress, ha->role stays CTRL_STANDBY and no WAL/term mutation occurs.

Contracts: Service [x] / State [x]

State Management

• State model: the PreVote tally (pre_votes_granted, pre_vote_replies, pre_vote_in_progress) lives in raft_state_t, mirroring the real-vote tally; quorum formula (n_peers / 2) + 1 reused from raft_on_vote_reply. The controller holds only a round-timeout handle + in-progress flag for cancellation.
• Concurrency: PreVote reply handling runs on the same single-threaded dispatch path as vote_reply_dispatch; no new lock introduced.

Implementation Notes

• Integration: PreVote success calls the unchanged ha_start_election_async/raft_start_election, so the real election (and bug-339/340 behavior) is unaffected.
• Validation: test_ha_prevote.c asserts the frozen-rejoin no-op (term constant, primary not demoted) and genuine-failover success.
• Risks: a failed PreVote leaving the watchdog un-armed (would freeze failover) — covered by an explicit re-arm test.

Error Handling

Error Strategy

• No PreVote majority (expected, frozen-rejoin or split): not an error — abandon cleanly, stay CTRL_STANDBY, re-arm liveness, log at debug/info with reason. The cluster term is unchanged.
• Round timeout: treated identically to "lost"; the next liveness cycle retries PreVote, succeeding once a majority's stickiness lapses (bounds legitimate failover by heartbeat-timeout, not unbounded — Req 5.4).
• Reply from a higher-term peer: the PreVote reply path does not adopt the peer's term (PreVote is side-effect-free); a genuinely higher term is observed and adopted only through the normal real-vote / heartbeat paths, unchanged.
• Unknown candidate id / malformed payload: deny PreVote (same as real-vote dispatch), log.

Monitoring

• Log every PreVote denial with candidate id + reason (primary, fresh_heartbeat, catch_up, stale_term, freshness) — Req 7.2.
• Log campaign abandonment distinctly from genuine failover so operators can tell a suppressed disruptive election from a real one — Req 7.3.

Testing Strategy

Unit Tests (raft library — test_raft.c)

• raft_on_pre_vote_request grants for a fresh, higher-term candidate when on_vote_eligible returns 1, and asserts term/voted_for/role unchanged (2.3, 6.2).
• raft_on_pre_vote_request denies on cand_term < term and when on_vote_eligible returns 0 (2.2, 3.x).
• raft_on_pre_vote_reply returns WON at quorum, LOST when all replied without quorum, with no role/term change (1.4, 1.5, 6.2).
• A granted PreVote does not set voted_for, so a later real vote in the same term is still grantable (2.4).

Integration Tests (controller — test_ha_prevote.c, test_ha_vote_gate.c)

• Frozen-then-resumed no-op: 3-node, primary alive; a resumed standby starts PreVote; both peers deny (PRIMARY + fresh heartbeat); assert no real RequestVote sent, cluster term unchanged, primary still CTRL_PRIMARY (4.1, 4.3, 4.4).
• Genuine failover: primary dead, peers' heartbeats stale; PreVote granted → real election → new primary within bounds (5.1, 5.4).
• Re-attach on heartbeat mid-round: PreVote in progress, primary heartbeat arrives → round cancelled, node re-attaches as standby, no term change (4.2).
• PreVote eligibility reuse: PRIMARY-deny, fresh-heartbeat-deny, catch-up-deny exercised through the PreVote path (3.1–3.3).
• Liveness re-arm after loss: failed PreVote re-arms the watchdog; a subsequent genuine failure still triggers (1.4 robustness).

Verification (Req 6)

• test_raft.c invariant assertions and a raft/raft.tla consistency argument (RN-1, tracked in research.md) confirm Election Safety and term monotonicity hold with the PreVote path; the real-vote path is unchanged so bug-339/340 coverage is preserved (5.3).