myxce/.kiro/specs/ha-prevote/design.md

# Design Document

## Overview

**Purpose.** This feature adds a **PreVote phase** to the XCE controller HA election path so that a controller which is merely temporarily unreachable cannot inflate its persistent term or unseat a healthy primary. Before a node increments its term and campaigns, it must first win a hypothetical PreVote majority — a "would you vote for me in term+1?" round that mutates no node's persistent term or recorded vote.

**Users.** Operators of 3-node controller HA clusters, and the controllers themselves, benefit: a frozen-then-resumed standby rejoins as a no-op instead of triggering the ~30s demote-and-restart re-election storm observed live on 2026-06-15.

**Impact.** The election trigger path changes from `trigger → real election (term++)` to `trigger → PreVote round → (only on majority) → real election (term++)`. The real-vote state machine, replication, and commit engine are unchanged. The change is additive: two new wire opcodes reusing the existing vote payload structs, pure PreVote decision **and tally** functions in the raft library (mirroring the production real-vote path `raft_on_vote_reply`), and PreVote orchestration in the controller that reuses the existing eligibility gate.

### Goals
- A node never increments its persistent term unless a PreVote majority is first obtained.
- A voter denies a PreVote whenever it still believes a leader is alive (leader stickiness), reusing the existing heartbeat-freshness predicate.
- A frozen-then-resumed standby in a healthy cluster gets no PreVote majority and re-attaches as a standby with the cluster term unchanged.
- Genuine leadership changes (true primary death, partition heal) still elect within existing timeout bounds.
- All Raft safety invariants and the bug-339/bug-340 fixes are preserved.

### Non-Goals
- Changes to replication, the commit/barrier engine, snapshot/WAL catch-up transfer, or config bootstrap.
- Redesign of the freshness / log-completeness comparison (reused as-is).
- Backward/rolling-upgrade compatibility (product is pre-release; all controllers run the same build).
- Leader-side CheckQuorum self-demotion and the defense-in-depth real-vote stickiness guard (see Architecture → Deferred decisions).
- **The startup / first-boot election** (the `ha_elect_ctx_t.done` semaphore path). PreVote gates only the **failover** trigger; on first boot there is no leader to protect and every node's heartbeat is stale, so a PreVote round would trivially pass. The startup election path is untouched.

## Boundary Commitments

### This Spec Owns
- The **PreVote round** on the election-trigger path: a side-effect-free majority probe that runs before any persistent term increment.
- The **voter-side PreVote decision**: term check + reuse of the existing eligibility gate + reuse of the dispatcher freshness gate, with **zero persistent mutation**.
- The two new wire opcodes (`preVoteRequest`, `preVoteReply`) and their dispatch.
- The trigger-path restructuring so that `truncate-WAL + CTRL_ELECTING` happens only after PreVote succeeds.
- PreVote observability (denial reasons, abandoned-campaign logging).

### Out of Boundary
- The real-vote state machine (`raft_on_vote_request`, `raft_on_vote_reply`, `raft_start_election`) — invoked unchanged after PreVote success.
- The freshness floor computation (`ha_vote_self_freshness`) and the eligibility gate (`raft_cb_vote_eligible`) — reused, not modified (except that the eligibility gate is now also called from the PreVote receiver path).
- Replication, commit barrier, snapshot transfer, demotion-convergence mechanics (truncate-above-committed + self-restart) — unchanged.
- Leader self-demotion / CheckQuorum.
- The startup-election `ha_elect_ctx_t.done` semaphore path — not gated by PreVote.

### Allowed Dependencies
- raft library callback seams (`raft_callbacks_t`), the raft-library vote-tally pattern (`raft_on_vote_reply` ⇒ `raft_on_pre_vote_reply`), `ha_heartbeat_is_fresh`, `ha_vote_self_freshness`, the capnp election transport (`xce_request` / vote endpoints / `peer_ctrl_msg_handler`), and the `xce_vote_request_hook` test seam.
- Dependency direction (must not be violated): **capnp/types → raft library (pure) → controller adapter (ha.c) → dispatch/transport (cluster.c)**. The raft library must not depend on controller state.

### Revalidation Triggers
- Any change to the `request_vote_req_t` / `request_vote_reply_t` payload shape (shared by real-vote and PreVote).
- Any change to `raft_cb_vote_eligible` semantics (now shared by both vote paths).
- Any change to the leader-stickiness predicate (`ha_heartbeat_is_fresh` / `heartbeat_timeout_ms`).
- Any change to the election-trigger entry points (`ha_on_primary_ctrl_fd_rdhup`, `liveness_poll_fn`).
- Introduction of mixed-version clusters (would reintroduce the deferred wire-compat concern).

## Architecture

### Existing Architecture Analysis

The election path is split across two layers (preserved by this design):

- **raft library** (`src/lib/raft/raft.c`, `raft.h`) — the canonical, TLA+-checkable state machine. Owns term, `voted_for`, role, and the pure vote-decision logic. Communicates outward only through `raft_callbacks_t` (no controller dependency). Today: `raft_start_election` increments+persists the term before any RPC; `raft_on_vote_request` adopts a higher term and steps down a leader in **Step 2 — before** the `on_vote_eligible` gate runs. This ordering is exactly why the existing heartbeat-stickiness gate cannot protect a leader from a fresh, higher-term RequestVote, and why the fix must be sender-side.
- **controller adapter** (`src/controller/ha.c`, `cluster.c`) — async transport + collection. `cluster.c` dispatches `OPCode_requestVote`, applies the freshness floor, maps candidate name→index, then calls the library. `ha.c` orchestrates `ha_start_election_async` (allocates `ha_elect_ctx_t`, self-vote, delegates to `raft_start_election`), collects replies (`handle_request_vote_reply` → `ha_elect_evaluate_and_signal`), and gates eligibility (`raft_cb_vote_eligible`: PRIMARY-deny, fresh-heartbeat-deny, catch-up-deny). Triggers: `ha_on_primary_ctrl_fd_rdhup` (RDHUP) and `liveness_poll_fn` (heartbeat timeout), which today transition to `CTRL_ELECTING`, truncate WAL, and start the election. (Note: `handle_request_vote_reply` is now the test-mode-only reply path; production reply collection — including endpoint/connect failures — routes through `vote_reply_dispatch`/`pre_vote_reply_dispatch` as the sole tally driver, per bug-1423/raft-adapter-conformance. `handle_request_vote_reply` is retained only for `XCE_TESTING` callers.)

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    subgraph trigger [Election Trigger]
        Rdhup[ha_on_primary_ctrl_fd_rdhup]
        Liveness[liveness_poll_fn]
    end
    subgraph controller [Controller Adapter ha.c cluster.c]
        BeginFailover[ha_begin_failover]
        PrevoteAsync[ha_start_prevote_async]
        ElectAsync[ha_start_election_async]
        Collect[raft_on_pre_vote_reply tally]
        DispReq[cluster.c preVoteRequest case]
        DispRep[cluster.c preVoteReply case]
        Eligible[raft_cb_vote_eligible reused]
        Fresh[ha_vote_self_freshness reused]
    end
    subgraph raftlib [Raft Library pure]
        Begin[raft_begin_pre_vote]
        OnReq[raft_on_pre_vote_request]
        OnRep[raft_on_pre_vote_reply]
        StartElection[raft_start_election unchanged]
    end
    subgraph wire [Transport]
        Capnp[preVoteRequest preVoteReply opcodes]
    end

    Rdhup --> BeginFailover
    Liveness --> BeginFailover
    BeginFailover --> PrevoteAsync
    PrevoteAsync --> Begin
    Begin --> Capnp
    Capnp --> DispReq
    DispReq --> Fresh
    DispReq --> Eligible
    DispReq --> OnReq
    OnReq --> Capnp
    Capnp --> DispRep
    DispRep --> OnRep
    OnRep --> Collect
    Collect -->|majority| ElectAsync
    ElectAsync --> StartElection
    Collect -->|lost or timeout| Rearm[re-arm liveness stay STANDBY]
```

**Architecture Integration**
- **Selected pattern**: side-effect-free probe interposed before an existing state transition (Raft dissertation §9.6 PreVote). Adopted, not invented.
- **Boundaries**: raft library owns the pure PreVote *decision and tally*; the controller owns the *orchestration/transport*. The voter eligibility gate is **shared** with the real-vote path via reuse, not duplication; the PreVote tally mirrors the production real-vote tally (`raft_on_vote_reply`) rather than the test-only `ha_elect_ctx_t` path.
- **Existing patterns preserved**: callback-seam isolation of the raft library; raft-library-owns-tally (`raft_on_vote_reply` ⇒ `raft_on_pre_vote_reply`); dispatcher-applies-freshness-before-library; `xce_vote_request_hook` test seam.
- **New components rationale**: a new opcode pair keeps the real-vote handler byte-for-byte unchanged (zero regression risk to bug-339/340); pure PreVote functions in the library keep the state machine TLA+-checkable.
- **Steering compliance**: async, non-blocking vtask handlers; long-lived connections; inbound connections vote-only; immutable-term discipline (PreVote mutates nothing).

### Deferred decisions (documented, not implemented)
- **Defense-in-depth real-vote stickiness guard** — rejecting a real higher-term RequestVote while a leader is believed alive. PreVote already guarantees a candidate only sends a real RequestVote after a majority's stickiness has lapsed, so in a same-version cluster no stickiness-violating real RequestVote can arise. Adding the guard would touch the safety-critical real-vote term-adoption path and risk regressing bug-339/340 for no in-scope benefit. Deferred; revisit only if mixed-version clusters or leader CheckQuorum are introduced.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Messaging / Events | Cap'n Proto (capnpc-c 1.0.1) | New `preVoteRequest`/`preVoteReply` opcodes reusing existing vote payload structs | No new structs; follows existing opcode-enum pattern |
| Backend / Services | C (raft library + controller) | Pure PreVote decision + tally functions; controller orchestration | No new libraries; reuses iomgr/delaymgr and the raft tally pattern |
| Infrastructure / Runtime | iomgr async transport, delaymgr | PreVote send/collect + round timeout | Non-blocking, mirrors election path |

## File Structure Plan

### Modified Files
- `src/schema/controller.capnp` — add two `OPCode` enum entries: `preVoteRequest`, `preVoteReply`. Reuse existing `RequestVoteRequest` / `RequestVoteReply` structs and their `Request.body` union arms (`requestvotemsg` / `requestvotereplymsg`); no new payload structs.
- `src/lib/raft/raft.h` — add pre-vote return codes (e.g. `RAFT_PRE_VOTE_WON`, `RAFT_PRE_VOTE_LOST`, `RAFT_PRE_VOTE_PENDING`), an optional `on_send_pre_vote_request` callback in `raft_callbacks_t`, pre-vote tally fields in `raft_state_t` (`pre_votes_granted`, `pre_vote_replies`, `pre_vote_in_progress`), and declarations for the three new functions.
- `src/lib/raft/raft.c` — add `raft_begin_pre_vote`, `raft_on_pre_vote_request`, `raft_on_pre_vote_reply` (all pure: no `on_persist`, no `term`/`voted_for`/`role` mutation). `raft_on_pre_vote_request` reuses `on_vote_eligible`. **`raft_on_pre_vote_reply` owns the PreVote tally in `raft_state_t` (mirroring `raft_on_vote_reply`) and returns `WON/LOST/PENDING` — it never fires `on_become_leader`/`on_become_follower`.**
- `src/controller/ha.h` — add a controller-side PreVote round handle on `ha_state_t` (round-timeout context + in-progress flag for cancellation). **No change to `ha_elect_ctx_t`** (the tally lives in `raft_state_t`, not here). (Note: as built, this handle is a module-static `prevote_round_ctx_t * _Atomic s_prevote_round_ctx` in `ha.c`, not a field added to `ha_state_t`/`ha.h` — `ha.h` was not modified.)
- `src/controller/ha.c` — add `ha_begin_failover` (common entry), `ha_start_prevote_async` (calls `raft_begin_pre_vote`, arms round timeout), and `pre_vote_reply_dispatch` (calls `raft_on_pre_vote_reply`, mirroring `vote_reply_dispatch → raft_on_vote_reply`); on `RAFT_PRE_VOTE_WON` perform truncate + `CTRL_ELECTING` + `ha_start_election_async`; on `RAFT_PRE_VOTE_LOST`/round-timeout stay `CTRL_STANDBY`, re-arm liveness, log abandonment. Modify `ha_on_primary_ctrl_fd_rdhup` and `liveness_poll_fn` to call `ha_begin_failover` (PreVote first; **no** truncate/`CTRL_ELECTING` until PreVote wins); modify `ha_on_heartbeat_received` to cancel an in-progress PreVote and re-attach.
- `src/controller/cluster.c` — add `OPCode_preVoteRequest` and `OPCode_preVoteReply` dispatch cases mirroring the real-vote cases, calling `raft_on_pre_vote_request` (no term adoption) and routing replies to `pre_vote_reply_dispatch`.

### New Files
- `src/controller/testsuite/test_ha_prevote.c` — PreVote gating + frozen-then-resumed no-op scenario + genuine-failover scenario.

### Test files extended
- `src/lib/testsuite/test_raft.c` — pure PreVote decision/tally unit tests + invariant assertions (no term mutation).
- `src/controller/testsuite/test_ha_vote_gate.c` — PreVote eligibility reuse of the role/heartbeat/catch-up gates.

## System Flows

### PreVote round (candidate side)

```mermaid
sequenceDiagram
    participant Trig as Trigger
    participant Cand as Candidate ha.c
    participant Peer as Peer voter
    Trig->>Cand: ha_begin_failover (stay CTRL_STANDBY)
    Cand->>Cand: ha_start_prevote_async (prospective term = current+1, no persist)
    Cand->>Peer: preVoteRequest term+1 tag lsn
    Note over Peer: stickiness gate then freshness gate then term check
    Peer-->>Cand: preVoteReply granted or denied (peer term unchanged)
    alt PreVote majority
        Cand->>Cand: now truncate WAL, CTRL_ELECTING
        Cand->>Cand: ha_start_election_async then raft_start_election (term++)
    else lost or round timeout
        Cand->>Cand: stay CTRL_STANDBY, re-arm liveness watchdog
        Note over Cand: next primary heartbeat re-attaches as standby
    end
```

Key decisions: the candidate remains `CTRL_STANDBY` and mutates nothing during the PreVote round; the destructive `truncate + CTRL_ELECTING` step is deferred to PreVote success. An incoming primary heartbeat during the round cancels it and re-attaches. The round is bounded by `election_timeout_ms`; on loss the watchdog is re-armed so a later genuine failure still triggers, and repeated PreVote attempts succeed once a majority's stickiness lapses.

### PreVote decision (voter side)

```mermaid
flowchart TD
    A[preVoteRequest received] --> B{role == CTRL_PRIMARY}
    B -- yes --> D[deny]
    B -- no --> C{role == CTRL_STANDBY and heartbeat fresh}
    C -- yes --> D
    C -- no --> E{in catch-up}
    E -- yes --> D
    E -- no --> F{req.term lt current_term}
    F -- yes --> D
    F -- no --> G{freshness floor ok}
    G -- no --> D
    G -- yes --> H[grant, no term change, no persist, no voted_for]
```

Steps B/C/E are exactly the existing `raft_cb_vote_eligible` gates (reused). Step G is exactly the existing dispatcher freshness floor (`ha_vote_self_freshness`). The only PreVote-specific rule is the **absence of any state mutation** on either grant or deny.

(Note: superseded by `raft-commit-safety`, see that spec's design.md — `raft_on_pre_vote_request` now also runs an internal `raft_log_ok(lastLogTerm, lastLogIndex)` check between step F and step B/C/E, denying stale-log candidates before the eligibility gate is reached; not shown in the flowchart above.)

## Requirements Traceability

| Requirement | Summary | Components | Interfaces / Flows |
|-------------|---------|------------|--------------------|
| 1.1 | PreVote before term increment on trigger | `ha_begin_failover`, `ha_start_prevote_async` | PreVote round flow |
| 1.2 | No term/vote mutation during round | `raft_begin_pre_vote`, `ha_start_prevote_async` | candidate stays STANDBY |
| 1.3 | Request prospective term+1, not persisted | `raft_begin_pre_vote` | preVoteRequest payload |
| 1.4 | No majority → no real election | `raft_on_pre_vote_reply` (LOST), `pre_vote_reply_dispatch` | round flow "lost" path |
| 1.5 | Majority → start real election | `ha_start_election_async`, `raft_start_election` (unchanged) | round flow "won" path |
| 2.1 | Grant only if real-vote freshness gate passes | `cluster.c preVoteRequest`, `ha_vote_self_freshness` | voter flow step G |
| 2.2 | Fail freshness → deny | same | voter flow step G |
| 2.3 | Grant/deny records no vote, no term | `raft_on_pre_vote_request` | voter flow leaf H |
| 2.4 | PreVote does not consume the real vote | `raft_on_pre_vote_request` (no `voted_for` write) | — |
| 3.1 | Deny while leader believed alive | `raft_cb_vote_eligible` (`ha_heartbeat_is_fresh`) | voter flow step C |
| 3.2 | Past self-election threshold → freshness-only eval | `raft_cb_vote_eligible` | voter flow C→E→F→G |
| 3.3 | Primary denies PreVotes | `raft_cb_vote_eligible` (PRIMARY-deny) | voter flow step B |
| 4.1 | Frozen-resumed gets no majority | full path | frozen-rejoin scenario test |
| 4.2 | Re-attach on next heartbeat, no term change | `ha_on_heartbeat_received` | round flow "lost"/cancel |
| 4.3 | Healthy primary not demoted/restarted | PreVote denies → no real RequestVote | — |
| 4.4 | Term unchanged, no re-election cycle | candidate stays STANDBY | — |
| 5.1 | True death → PreVotes granted → elect | `raft_cb_vote_eligible` (stale heartbeat → eligible) | genuine-failover test |
| 5.2 | Partition heal → adopt higher term | `raft_on_vote_reply` (unchanged) | real-vote path |
| 5.3 | No regression of bug-339/bug-340 | real-vote path untouched | — |
| 5.4 | Complete within election-timeout bounds | `election_timeout_ms` round timeout | round flow |
| 6.1 | Election Safety preserved | real-vote path untouched; PreVote grants no leadership | RN-1 (research.md) |
| 6.2 | Term monotonicity; PreVote never changes term | `raft_on_pre_vote_*` (pure) | — |
| 6.3 | Leader Completeness, monotonic commit index | freshness floor reused; commit path untouched | — |
| 6.4 | Consistent with raft.tla + checklist | `test_raft.c` invariant tests | RN-1 |
| 7.1 | PreVote over existing transport, no disruption to real votes | new opcodes, reused arms | — |
| 7.2 | Log PreVote denials with candidate + reason | `cluster.c preVoteRequest`, `on_trace`/`tlog` | — |
| 7.3 | Log abandoned campaign | `pre_vote_reply_dispatch` (LOST/timeout) | round flow |

## Components and Interfaces

| Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|-------|--------|--------------|--------------------------|-----------|
| raft pre-vote functions | raft library | Pure PreVote decision + tally | 1.2, 1.3, 2.3, 2.4, 6.2 | `on_vote_eligible` (P0), `on_send_pre_vote_request` (P1) | Service, State |
| PreVote dispatch | cluster.c | Receive/answer PreVote, no mutation | 2.1, 2.2, 3.1–3.3, 7.1, 7.2 | `ha_vote_self_freshness` (P0), `raft_on_pre_vote_request` (P0) | Service, Event |
| PreVote orchestration | ha.c | Run round, gate trigger, branch to election | 1.1, 1.4, 1.5, 4.x, 5.4, 7.3 | `raft_on_pre_vote_reply` (P0), `ha_start_election_async` (P0), `liveness` (P1) | Service, State |
| Trigger restructuring | ha.c | Defer truncate/CTRL_ELECTING until PreVote wins | 1.1, 4.2, 4.3, 4.4 | `ha_on_primary_ctrl_fd_rdhup`, `liveness_poll_fn` (P0) | State |

### Raft library — pre-vote functions

| Field | Detail |
|-------|--------|
| Intent | Pure PreVote decision and tally; canonical, TLA+-checkable; mutates no persistent state |
| Requirements | 1.2, 1.3, 2.3, 2.4, 6.2 |

**Responsibilities & Constraints**
- `raft_begin_pre_vote`: compute prospective term = `term + 1` **without** assigning it to `s->term` or persisting; set `pre_vote_in_progress`, `pre_votes_granted = 1` (self); emit `on_send_pre_vote_request(ctx, peer_idx, prospective_term)` per peer.
- `raft_on_pre_vote_request`: deny if `cand_term < s->term`; else return the result of `on_vote_eligible(ctx, from_idx, cand_term)`; **never** touches `term`, `voted_for`, `role`, or calls `on_persist`. Freshness is applied by the dispatcher before this call (same contract as real vote). (Note: superseded by `raft-commit-safety`, see that spec's design.md — that spec (closed 2026-06-25, i.e. before this spec's own close) added a `raft_log_ok(lastLogTerm, lastLogIndex)` gate as an internal Step 2 inside `raft_on_pre_vote_request` itself, run **after** the stale-term check and **before** `on_vote_eligible`. The function signature grew two params — `cand_last_log_term`, `cand_last_log_index` — see the corrected Service Interface below. The dispatcher's `ha_vote_self_freshness` floor check in `cluster.c` still also runs, so freshness is now enforced at both layers.)
- `raft_on_pre_vote_reply`: **owns the PreVote tally** in `raft_state_t` (`pre_votes_granted`, `pre_vote_replies`), mirroring `raft_on_vote_reply`; return `RAFT_PRE_VOTE_WON` at quorum, `RAFT_PRE_VOTE_LOST` when all replied without quorum, else `RAFT_PRE_VOTE_PENDING`. Unlike `raft_on_vote_reply`, it **never** fires `on_become_leader`/`on_become_follower` and makes no role/term change — the controller acts on the returned status.
- **Invariant**: across all three functions, `s->term`, `s->voted_for`, `s->voted_for_term`, and `s->role` are unchanged, and `on_persist` is never invoked.

**Contracts**: Service [x] / State [x]

##### Service Interface
```c
/* All pure: no persist, no term/role/voted_for mutation. */
int raft_begin_pre_vote(raft_state_t *s);                 /* sends pre-votes, self-grant in raft_state_t */
int raft_on_pre_vote_request(raft_state_t *s, int from_idx, uint64_t cand_term,
                             uint64_t cand_last_log_term,
                             uint64_t cand_last_log_index);  /* RAFT_GRANT=1 / RAFT_DENY=0 */
int raft_on_pre_vote_reply(raft_state_t *s,
                           int peer_idx, int granted);    /* tally in raft_state_t -> WON / LOST / PENDING */
```
(Note: superseded by `raft-commit-safety` — `raft_on_pre_vote_request` gained the `cand_last_log_term`/`cand_last_log_index` params and an internal `raft_log_ok` freshness check; see the Responsibilities & Constraints note above.)
- Preconditions: `raft_begin_pre_vote` requires `role == RAFT_FOLLOWER`.
- Postconditions: on `RAFT_PRE_VOTE_WON` the caller invokes the unchanged `raft_start_election`.
- Invariants: persistent state untouched (asserted in `test_raft.c`).

**Implementation Notes**
- Integration: `on_vote_eligible` is the existing `raft_cb_vote_eligible`; reusing it gives PRIMARY-deny + heartbeat-stickiness + catch-up-deny for free (Req 3.x).
- Validation: unit tests assert `term`/`voted_for` unchanged after a granted PreVote.
- Risks: accidental term mutation — mitigated by keeping all term writes in `raft_start_election` only.

### Controller — PreVote dispatch (cluster.c)

| Field | Detail |
|-------|--------|
| Intent | Answer `preVoteRequest` with freshness + stickiness gates, no mutation; route `preVoteReply` to the tally |
| Requirements | 2.1, 2.2, 3.1–3.3, 7.1, 7.2 |

**Responsibilities & Constraints**
- `OPCode_preVoteRequest`: parse `request_vote_req_t`; compute freshness floor via `ha_vote_self_freshness`; deny on freshness failure; otherwise call `raft_on_pre_vote_request`; reply `request_vote_reply_t` with **unchanged** `term`, the grant bit, and `peer_config_tag`/`peer_last_log_lsn`. Log denials with candidate id + reason (Req 7.2).
- `OPCode_preVoteReply`: route to `pre_vote_reply_dispatch` → `raft_on_pre_vote_reply` (raft-library tally), paralleling the production `vote_reply_dispatch` → `raft_on_vote_reply`.
- Inbound connections remain vote-only; the reply carries no state push.

**Contracts**: Service [x] / Event [x]

##### Event Contract
- Published: `preVoteRequest` (term+1, candidate_id, config_tag, last_log_lsn).
- Subscribed: `preVoteReply` (term, vote_granted, peer_config_tag, peer_last_log_lsn).
- Delivery: best-effort over the existing election transport; missing/denied replies count as non-grants. No ordering requirement beyond per-peer idempotency: `raft_on_pre_vote_reply` receives `peer_idx` and tracks a per-peer replied flag in `raft_state_t` to avoid double-counting duplicate replies.

### Controller — PreVote orchestration & trigger restructuring (ha.c)

| Field | Detail |
|-------|--------|
| Intent | Run the side-effect-free round, gate the failover trigger, collect replies, branch to real election or clean abandon |
| Requirements | 1.1, 1.4, 1.5, 4.1–4.4, 5.4, 7.3 |

**Responsibilities & Constraints**
- `ha_begin_failover`: single entry from both `ha_on_primary_ctrl_fd_rdhup` and `liveness_poll_fn`; starts `ha_start_prevote_async`; **does not** truncate WAL or transition to `CTRL_ELECTING` (deferred to PreVote success).
- `ha_start_prevote_async`: call `raft_begin_pre_vote` (self-grant + send `preVoteRequest` per peer via `on_send_pre_vote_request`); arm an `election_timeout_ms` round timeout. The PreVote tally lives in `raft_state_t`, not a controller struct.
- `pre_vote_reply_dispatch` → `raft_on_pre_vote_reply`: on `RAFT_PRE_VOTE_WON` → truncate + `CTRL_ELECTING` + `ha_start_election_async`; on `RAFT_PRE_VOTE_LOST` or round timeout → stay `CTRL_STANDBY`, re-arm liveness, log abandonment (Req 7.3).
- `ha_on_heartbeat_received`: if a PreVote round is in progress, cancel it (clear the round handle) and re-attach as standby (Req 4.2) — no term change.
- **Invariant**: while a PreVote round is in progress, `ha->role` stays `CTRL_STANDBY` and no WAL/term mutation occurs.

**Contracts**: Service [x] / State [x]

##### State Management
- State model: the PreVote tally (`pre_votes_granted`, `pre_vote_replies`, `pre_vote_in_progress`) lives in `raft_state_t`, mirroring the real-vote tally; quorum formula `(n_peers / 2) + 1` reused from `raft_on_vote_reply`. The controller holds only a round-timeout handle + in-progress flag for cancellation.
- Concurrency: PreVote reply handling runs on the same single-threaded dispatch path as `vote_reply_dispatch`; no new lock introduced.

**Implementation Notes**
- Integration: PreVote success calls the **unchanged** `ha_start_election_async`/`raft_start_election`, so the real election (and bug-339/340 behavior) is unaffected.
- Validation: `test_ha_prevote.c` asserts the frozen-rejoin no-op (term constant, primary not demoted) and genuine-failover success.
- Risks: a failed PreVote leaving the watchdog un-armed (would freeze failover) — covered by an explicit re-arm test.

## Error Handling

### Error Strategy
- **No PreVote majority** (expected, frozen-rejoin or split): not an error — abandon cleanly, stay `CTRL_STANDBY`, re-arm liveness, log at debug/info with reason. The cluster term is unchanged.
- **Round timeout**: treated identically to "lost"; the next liveness cycle retries PreVote, succeeding once a majority's stickiness lapses (bounds legitimate failover by heartbeat-timeout, not unbounded — Req 5.4).
- **Reply from a higher-term peer**: the PreVote reply path does **not** adopt the peer's term (PreVote is side-effect-free); a genuinely higher term is observed and adopted only through the normal real-vote / heartbeat paths, unchanged.
- **Unknown candidate id / malformed payload**: deny PreVote (same as real-vote dispatch), log.

### Monitoring
- Log every PreVote denial with candidate id + reason (`primary`, `fresh_heartbeat`, `catch_up`, `stale_term`, `freshness`) — Req 7.2.
- Log campaign abandonment distinctly from genuine failover so operators can tell a suppressed disruptive election from a real one — Req 7.3.

## Testing Strategy

### Unit Tests (raft library — `test_raft.c`)
- `raft_on_pre_vote_request` grants for a fresh, higher-term candidate when `on_vote_eligible` returns 1, and asserts `term`/`voted_for`/`role` unchanged (2.3, 6.2).
- `raft_on_pre_vote_request` denies on `cand_term < term` and when `on_vote_eligible` returns 0 (2.2, 3.x).
- `raft_on_pre_vote_reply` returns WON at quorum, LOST when all replied without quorum, with no role/term change (1.4, 1.5, 6.2).
- A granted PreVote does not set `voted_for`, so a later real vote in the same term is still grantable (2.4).

### Integration Tests (controller — `test_ha_prevote.c`, `test_ha_vote_gate.c`)
- **Frozen-then-resumed no-op**: 3-node, primary alive; a resumed standby starts PreVote; both peers deny (PRIMARY + fresh heartbeat); assert no real RequestVote sent, cluster term unchanged, primary still `CTRL_PRIMARY` (4.1, 4.3, 4.4).
- **Genuine failover**: primary dead, peers' heartbeats stale; PreVote granted → real election → new primary within bounds (5.1, 5.4).
- **Re-attach on heartbeat mid-round**: PreVote in progress, primary heartbeat arrives → round cancelled, node re-attaches as standby, no term change (4.2).
- **PreVote eligibility reuse**: PRIMARY-deny, fresh-heartbeat-deny, catch-up-deny exercised through the PreVote path (3.1–3.3).
- **Liveness re-arm after loss**: failed PreVote re-arms the watchdog; a subsequent genuine failure still triggers (1.4 robustness).

### Verification (Req 6)
- `test_raft.c` invariant assertions and a `raft/raft.tla` consistency argument (RN-1, tracked in `research.md`) confirm Election Safety and term monotonicity hold with the PreVote path; the real-vote path is unchanged so bug-339/340 coverage is preserved (5.3).