# Design Document ## Overview **Purpose.** This feature adds a **PreVote phase** to the XCE controller HA election path so that a controller which is merely temporarily unreachable cannot inflate its persistent term or unseat a healthy primary. Before a node increments its term and campaigns, it must first win a hypothetical PreVote majority — a "would you vote for me in term+1?" round that mutates no node's persistent term or recorded vote. **Users.** Operators of 3-node controller HA clusters, and the controllers themselves, benefit: a frozen-then-resumed standby rejoins as a no-op instead of triggering the ~30s demote-and-restart re-election storm observed live on 2026-06-15. **Impact.** The election trigger path changes from `trigger → real election (term++)` to `trigger → PreVote round → (only on majority) → real election (term++)`. The real-vote state machine, replication, and commit engine are unchanged. The change is additive: two new wire opcodes reusing the existing vote payload structs, pure PreVote decision **and tally** functions in the raft library (mirroring the production real-vote path `raft_on_vote_reply`), and PreVote orchestration in the controller that reuses the existing eligibility gate. ### Goals - A node never increments its persistent term unless a PreVote majority is first obtained. - A voter denies a PreVote whenever it still believes a leader is alive (leader stickiness), reusing the existing heartbeat-freshness predicate. - A frozen-then-resumed standby in a healthy cluster gets no PreVote majority and re-attaches as a standby with the cluster term unchanged. - Genuine leadership changes (true primary death, partition heal) still elect within existing timeout bounds. - All Raft safety invariants and the bug-339/bug-340 fixes are preserved. ### Non-Goals - Changes to replication, the commit/barrier engine, snapshot/WAL catch-up transfer, or config bootstrap. - Redesign of the freshness / log-completeness comparison (reused as-is). - Backward/rolling-upgrade compatibility (product is pre-release; all controllers run the same build). - Leader-side CheckQuorum self-demotion and the defense-in-depth real-vote stickiness guard (see Architecture → Deferred decisions). - **The startup / first-boot election** (the `ha_elect_ctx_t.done` semaphore path). PreVote gates only the **failover** trigger; on first boot there is no leader to protect and every node's heartbeat is stale, so a PreVote round would trivially pass. The startup election path is untouched. ## Boundary Commitments ### This Spec Owns - The **PreVote round** on the election-trigger path: a side-effect-free majority probe that runs before any persistent term increment. - The **voter-side PreVote decision**: term check + reuse of the existing eligibility gate + reuse of the dispatcher freshness gate, with **zero persistent mutation**. - The two new wire opcodes (`preVoteRequest`, `preVoteReply`) and their dispatch. - The trigger-path restructuring so that `truncate-WAL + CTRL_ELECTING` happens only after PreVote succeeds. - PreVote observability (denial reasons, abandoned-campaign logging). ### Out of Boundary - The real-vote state machine (`raft_on_vote_request`, `raft_on_vote_reply`, `raft_start_election`) — invoked unchanged after PreVote success. - The freshness floor computation (`ha_vote_self_freshness`) and the eligibility gate (`raft_cb_vote_eligible`) — reused, not modified (except that the eligibility gate is now also called from the PreVote receiver path). - Replication, commit barrier, snapshot transfer, demotion-convergence mechanics (truncate-above-committed + self-restart) — unchanged. - Leader self-demotion / CheckQuorum. - The startup-election `ha_elect_ctx_t.done` semaphore path — not gated by PreVote. ### Allowed Dependencies - raft library callback seams (`raft_callbacks_t`), the raft-library vote-tally pattern (`raft_on_vote_reply` ⇒ `raft_on_pre_vote_reply`), `ha_heartbeat_is_fresh`, `ha_vote_self_freshness`, the capnp election transport (`xce_request` / vote endpoints / `peer_ctrl_msg_handler`), and the `xce_vote_request_hook` test seam. - Dependency direction (must not be violated): **capnp/types → raft library (pure) → controller adapter (ha.c) → dispatch/transport (cluster.c)**. The raft library must not depend on controller state. ### Revalidation Triggers - Any change to the `request_vote_req_t` / `request_vote_reply_t` payload shape (shared by real-vote and PreVote). - Any change to `raft_cb_vote_eligible` semantics (now shared by both vote paths). - Any change to the leader-stickiness predicate (`ha_heartbeat_is_fresh` / `heartbeat_timeout_ms`). - Any change to the election-trigger entry points (`ha_on_primary_ctrl_fd_rdhup`, `liveness_poll_fn`). - Introduction of mixed-version clusters (would reintroduce the deferred wire-compat concern). ## Architecture ### Existing Architecture Analysis The election path is split across two layers (preserved by this design): - **raft library** (`src/lib/raft/raft.c`, `raft.h`) — the canonical, TLA+-checkable state machine. Owns term, `voted_for`, role, and the pure vote-decision logic. Communicates outward only through `raft_callbacks_t` (no controller dependency). Today: `raft_start_election` increments+persists the term before any RPC; `raft_on_vote_request` adopts a higher term and steps down a leader in **Step 2 — before** the `on_vote_eligible` gate runs. This ordering is exactly why the existing heartbeat-stickiness gate cannot protect a leader from a fresh, higher-term RequestVote, and why the fix must be sender-side. - **controller adapter** (`src/controller/ha.c`, `cluster.c`) — async transport + collection. `cluster.c` dispatches `OPCode_requestVote`, applies the freshness floor, maps candidate name→index, then calls the library. `ha.c` orchestrates `ha_start_election_async` (allocates `ha_elect_ctx_t`, self-vote, delegates to `raft_start_election`), collects replies (`handle_request_vote_reply` → `ha_elect_evaluate_and_signal`), and gates eligibility (`raft_cb_vote_eligible`: PRIMARY-deny, fresh-heartbeat-deny, catch-up-deny). Triggers: `ha_on_primary_ctrl_fd_rdhup` (RDHUP) and `liveness_poll_fn` (heartbeat timeout), which today transition to `CTRL_ELECTING`, truncate WAL, and start the election. (Note: `handle_request_vote_reply` is now the test-mode-only reply path; production reply collection — including endpoint/connect failures — routes through `vote_reply_dispatch`/`pre_vote_reply_dispatch` as the sole tally driver, per bug-1423/raft-adapter-conformance. `handle_request_vote_reply` is retained only for `XCE_TESTING` callers.) ### Architecture Pattern & Boundary Map ```mermaid graph TB subgraph trigger [Election Trigger] Rdhup[ha_on_primary_ctrl_fd_rdhup] Liveness[liveness_poll_fn] end subgraph controller [Controller Adapter ha.c cluster.c] BeginFailover[ha_begin_failover] PrevoteAsync[ha_start_prevote_async] ElectAsync[ha_start_election_async] Collect[raft_on_pre_vote_reply tally] DispReq[cluster.c preVoteRequest case] DispRep[cluster.c preVoteReply case] Eligible[raft_cb_vote_eligible reused] Fresh[ha_vote_self_freshness reused] end subgraph raftlib [Raft Library pure] Begin[raft_begin_pre_vote] OnReq[raft_on_pre_vote_request] OnRep[raft_on_pre_vote_reply] StartElection[raft_start_election unchanged] end subgraph wire [Transport] Capnp[preVoteRequest preVoteReply opcodes] end Rdhup --> BeginFailover Liveness --> BeginFailover BeginFailover --> PrevoteAsync PrevoteAsync --> Begin Begin --> Capnp Capnp --> DispReq DispReq --> Fresh DispReq --> Eligible DispReq --> OnReq OnReq --> Capnp Capnp --> DispRep DispRep --> OnRep OnRep --> Collect Collect -->|majority| ElectAsync ElectAsync --> StartElection Collect -->|lost or timeout| Rearm[re-arm liveness stay STANDBY] ``` **Architecture Integration** - **Selected pattern**: side-effect-free probe interposed before an existing state transition (Raft dissertation §9.6 PreVote). Adopted, not invented. - **Boundaries**: raft library owns the pure PreVote *decision and tally*; the controller owns the *orchestration/transport*. The voter eligibility gate is **shared** with the real-vote path via reuse, not duplication; the PreVote tally mirrors the production real-vote tally (`raft_on_vote_reply`) rather than the test-only `ha_elect_ctx_t` path. - **Existing patterns preserved**: callback-seam isolation of the raft library; raft-library-owns-tally (`raft_on_vote_reply` ⇒ `raft_on_pre_vote_reply`); dispatcher-applies-freshness-before-library; `xce_vote_request_hook` test seam. - **New components rationale**: a new opcode pair keeps the real-vote handler byte-for-byte unchanged (zero regression risk to bug-339/340); pure PreVote functions in the library keep the state machine TLA+-checkable. - **Steering compliance**: async, non-blocking vtask handlers; long-lived connections; inbound connections vote-only; immutable-term discipline (PreVote mutates nothing). ### Deferred decisions (documented, not implemented) - **Defense-in-depth real-vote stickiness guard** — rejecting a real higher-term RequestVote while a leader is believed alive. PreVote already guarantees a candidate only sends a real RequestVote after a majority's stickiness has lapsed, so in a same-version cluster no stickiness-violating real RequestVote can arise. Adding the guard would touch the safety-critical real-vote term-adoption path and risk regressing bug-339/340 for no in-scope benefit. Deferred; revisit only if mixed-version clusters or leader CheckQuorum are introduced. ### Technology Stack | Layer | Choice / Version | Role in Feature | Notes | |-------|------------------|-----------------|-------| | Messaging / Events | Cap'n Proto (capnpc-c 1.0.1) | New `preVoteRequest`/`preVoteReply` opcodes reusing existing vote payload structs | No new structs; follows existing opcode-enum pattern | | Backend / Services | C (raft library + controller) | Pure PreVote decision + tally functions; controller orchestration | No new libraries; reuses iomgr/delaymgr and the raft tally pattern | | Infrastructure / Runtime | iomgr async transport, delaymgr | PreVote send/collect + round timeout | Non-blocking, mirrors election path | ## File Structure Plan ### Modified Files - `src/schema/controller.capnp` — add two `OPCode` enum entries: `preVoteRequest`, `preVoteReply`. Reuse existing `RequestVoteRequest` / `RequestVoteReply` structs and their `Request.body` union arms (`requestvotemsg` / `requestvotereplymsg`); no new payload structs. - `src/lib/raft/raft.h` — add pre-vote return codes (e.g. `RAFT_PRE_VOTE_WON`, `RAFT_PRE_VOTE_LOST`, `RAFT_PRE_VOTE_PENDING`), an optional `on_send_pre_vote_request` callback in `raft_callbacks_t`, pre-vote tally fields in `raft_state_t` (`pre_votes_granted`, `pre_vote_replies`, `pre_vote_in_progress`), and declarations for the three new functions. - `src/lib/raft/raft.c` — add `raft_begin_pre_vote`, `raft_on_pre_vote_request`, `raft_on_pre_vote_reply` (all pure: no `on_persist`, no `term`/`voted_for`/`role` mutation). `raft_on_pre_vote_request` reuses `on_vote_eligible`. **`raft_on_pre_vote_reply` owns the PreVote tally in `raft_state_t` (mirroring `raft_on_vote_reply`) and returns `WON/LOST/PENDING` — it never fires `on_become_leader`/`on_become_follower`.** - `src/controller/ha.h` — add a controller-side PreVote round handle on `ha_state_t` (round-timeout context + in-progress flag for cancellation). **No change to `ha_elect_ctx_t`** (the tally lives in `raft_state_t`, not here). (Note: as built, this handle is a module-static `prevote_round_ctx_t * _Atomic s_prevote_round_ctx` in `ha.c`, not a field added to `ha_state_t`/`ha.h` — `ha.h` was not modified.) - `src/controller/ha.c` — add `ha_begin_failover` (common entry), `ha_start_prevote_async` (calls `raft_begin_pre_vote`, arms round timeout), and `pre_vote_reply_dispatch` (calls `raft_on_pre_vote_reply`, mirroring `vote_reply_dispatch → raft_on_vote_reply`); on `RAFT_PRE_VOTE_WON` perform truncate + `CTRL_ELECTING` + `ha_start_election_async`; on `RAFT_PRE_VOTE_LOST`/round-timeout stay `CTRL_STANDBY`, re-arm liveness, log abandonment. Modify `ha_on_primary_ctrl_fd_rdhup` and `liveness_poll_fn` to call `ha_begin_failover` (PreVote first; **no** truncate/`CTRL_ELECTING` until PreVote wins); modify `ha_on_heartbeat_received` to cancel an in-progress PreVote and re-attach. - `src/controller/cluster.c` — add `OPCode_preVoteRequest` and `OPCode_preVoteReply` dispatch cases mirroring the real-vote cases, calling `raft_on_pre_vote_request` (no term adoption) and routing replies to `pre_vote_reply_dispatch`. ### New Files - `src/controller/testsuite/test_ha_prevote.c` — PreVote gating + frozen-then-resumed no-op scenario + genuine-failover scenario. ### Test files extended - `src/lib/testsuite/test_raft.c` — pure PreVote decision/tally unit tests + invariant assertions (no term mutation). - `src/controller/testsuite/test_ha_vote_gate.c` — PreVote eligibility reuse of the role/heartbeat/catch-up gates. ## System Flows ### PreVote round (candidate side) ```mermaid sequenceDiagram participant Trig as Trigger participant Cand as Candidate ha.c participant Peer as Peer voter Trig->>Cand: ha_begin_failover (stay CTRL_STANDBY) Cand->>Cand: ha_start_prevote_async (prospective term = current+1, no persist) Cand->>Peer: preVoteRequest term+1 tag lsn Note over Peer: stickiness gate then freshness gate then term check Peer-->>Cand: preVoteReply granted or denied (peer term unchanged) alt PreVote majority Cand->>Cand: now truncate WAL, CTRL_ELECTING Cand->>Cand: ha_start_election_async then raft_start_election (term++) else lost or round timeout Cand->>Cand: stay CTRL_STANDBY, re-arm liveness watchdog Note over Cand: next primary heartbeat re-attaches as standby end ``` Key decisions: the candidate remains `CTRL_STANDBY` and mutates nothing during the PreVote round; the destructive `truncate + CTRL_ELECTING` step is deferred to PreVote success. An incoming primary heartbeat during the round cancels it and re-attaches. The round is bounded by `election_timeout_ms`; on loss the watchdog is re-armed so a later genuine failure still triggers, and repeated PreVote attempts succeed once a majority's stickiness lapses. ### PreVote decision (voter side) ```mermaid flowchart TD A[preVoteRequest received] --> B{role == CTRL_PRIMARY} B -- yes --> D[deny] B -- no --> C{role == CTRL_STANDBY and heartbeat fresh} C -- yes --> D C -- no --> E{in catch-up} E -- yes --> D E -- no --> F{req.term lt current_term} F -- yes --> D F -- no --> G{freshness floor ok} G -- no --> D G -- yes --> H[grant, no term change, no persist, no voted_for] ``` Steps B/C/E are exactly the existing `raft_cb_vote_eligible` gates (reused). Step G is exactly the existing dispatcher freshness floor (`ha_vote_self_freshness`). The only PreVote-specific rule is the **absence of any state mutation** on either grant or deny. (Note: superseded by `raft-commit-safety`, see that spec's design.md — `raft_on_pre_vote_request` now also runs an internal `raft_log_ok(lastLogTerm, lastLogIndex)` check between step F and step B/C/E, denying stale-log candidates before the eligibility gate is reached; not shown in the flowchart above.) ## Requirements Traceability | Requirement | Summary | Components | Interfaces / Flows | |-------------|---------|------------|--------------------| | 1.1 | PreVote before term increment on trigger | `ha_begin_failover`, `ha_start_prevote_async` | PreVote round flow | | 1.2 | No term/vote mutation during round | `raft_begin_pre_vote`, `ha_start_prevote_async` | candidate stays STANDBY | | 1.3 | Request prospective term+1, not persisted | `raft_begin_pre_vote` | preVoteRequest payload | | 1.4 | No majority → no real election | `raft_on_pre_vote_reply` (LOST), `pre_vote_reply_dispatch` | round flow "lost" path | | 1.5 | Majority → start real election | `ha_start_election_async`, `raft_start_election` (unchanged) | round flow "won" path | | 2.1 | Grant only if real-vote freshness gate passes | `cluster.c preVoteRequest`, `ha_vote_self_freshness` | voter flow step G | | 2.2 | Fail freshness → deny | same | voter flow step G | | 2.3 | Grant/deny records no vote, no term | `raft_on_pre_vote_request` | voter flow leaf H | | 2.4 | PreVote does not consume the real vote | `raft_on_pre_vote_request` (no `voted_for` write) | — | | 3.1 | Deny while leader believed alive | `raft_cb_vote_eligible` (`ha_heartbeat_is_fresh`) | voter flow step C | | 3.2 | Past self-election threshold → freshness-only eval | `raft_cb_vote_eligible` | voter flow C→E→F→G | | 3.3 | Primary denies PreVotes | `raft_cb_vote_eligible` (PRIMARY-deny) | voter flow step B | | 4.1 | Frozen-resumed gets no majority | full path | frozen-rejoin scenario test | | 4.2 | Re-attach on next heartbeat, no term change | `ha_on_heartbeat_received` | round flow "lost"/cancel | | 4.3 | Healthy primary not demoted/restarted | PreVote denies → no real RequestVote | — | | 4.4 | Term unchanged, no re-election cycle | candidate stays STANDBY | — | | 5.1 | True death → PreVotes granted → elect | `raft_cb_vote_eligible` (stale heartbeat → eligible) | genuine-failover test | | 5.2 | Partition heal → adopt higher term | `raft_on_vote_reply` (unchanged) | real-vote path | | 5.3 | No regression of bug-339/bug-340 | real-vote path untouched | — | | 5.4 | Complete within election-timeout bounds | `election_timeout_ms` round timeout | round flow | | 6.1 | Election Safety preserved | real-vote path untouched; PreVote grants no leadership | RN-1 (research.md) | | 6.2 | Term monotonicity; PreVote never changes term | `raft_on_pre_vote_*` (pure) | — | | 6.3 | Leader Completeness, monotonic commit index | freshness floor reused; commit path untouched | — | | 6.4 | Consistent with raft.tla + checklist | `test_raft.c` invariant tests | RN-1 | | 7.1 | PreVote over existing transport, no disruption to real votes | new opcodes, reused arms | — | | 7.2 | Log PreVote denials with candidate + reason | `cluster.c preVoteRequest`, `on_trace`/`tlog` | — | | 7.3 | Log abandoned campaign | `pre_vote_reply_dispatch` (LOST/timeout) | round flow | ## Components and Interfaces | Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | |-----------|-------|--------|--------------|--------------------------|-----------| | raft pre-vote functions | raft library | Pure PreVote decision + tally | 1.2, 1.3, 2.3, 2.4, 6.2 | `on_vote_eligible` (P0), `on_send_pre_vote_request` (P1) | Service, State | | PreVote dispatch | cluster.c | Receive/answer PreVote, no mutation | 2.1, 2.2, 3.1–3.3, 7.1, 7.2 | `ha_vote_self_freshness` (P0), `raft_on_pre_vote_request` (P0) | Service, Event | | PreVote orchestration | ha.c | Run round, gate trigger, branch to election | 1.1, 1.4, 1.5, 4.x, 5.4, 7.3 | `raft_on_pre_vote_reply` (P0), `ha_start_election_async` (P0), `liveness` (P1) | Service, State | | Trigger restructuring | ha.c | Defer truncate/CTRL_ELECTING until PreVote wins | 1.1, 4.2, 4.3, 4.4 | `ha_on_primary_ctrl_fd_rdhup`, `liveness_poll_fn` (P0) | State | ### Raft library — pre-vote functions | Field | Detail | |-------|--------| | Intent | Pure PreVote decision and tally; canonical, TLA+-checkable; mutates no persistent state | | Requirements | 1.2, 1.3, 2.3, 2.4, 6.2 | **Responsibilities & Constraints** - `raft_begin_pre_vote`: compute prospective term = `term + 1` **without** assigning it to `s->term` or persisting; set `pre_vote_in_progress`, `pre_votes_granted = 1` (self); emit `on_send_pre_vote_request(ctx, peer_idx, prospective_term)` per peer. - `raft_on_pre_vote_request`: deny if `cand_term < s->term`; else return the result of `on_vote_eligible(ctx, from_idx, cand_term)`; **never** touches `term`, `voted_for`, `role`, or calls `on_persist`. Freshness is applied by the dispatcher before this call (same contract as real vote). (Note: superseded by `raft-commit-safety`, see that spec's design.md — that spec (closed 2026-06-25, i.e. before this spec's own close) added a `raft_log_ok(lastLogTerm, lastLogIndex)` gate as an internal Step 2 inside `raft_on_pre_vote_request` itself, run **after** the stale-term check and **before** `on_vote_eligible`. The function signature grew two params — `cand_last_log_term`, `cand_last_log_index` — see the corrected Service Interface below. The dispatcher's `ha_vote_self_freshness` floor check in `cluster.c` still also runs, so freshness is now enforced at both layers.) - `raft_on_pre_vote_reply`: **owns the PreVote tally** in `raft_state_t` (`pre_votes_granted`, `pre_vote_replies`), mirroring `raft_on_vote_reply`; return `RAFT_PRE_VOTE_WON` at quorum, `RAFT_PRE_VOTE_LOST` when all replied without quorum, else `RAFT_PRE_VOTE_PENDING`. Unlike `raft_on_vote_reply`, it **never** fires `on_become_leader`/`on_become_follower` and makes no role/term change — the controller acts on the returned status. - **Invariant**: across all three functions, `s->term`, `s->voted_for`, `s->voted_for_term`, and `s->role` are unchanged, and `on_persist` is never invoked. **Contracts**: Service [x] / State [x] ##### Service Interface ```c /* All pure: no persist, no term/role/voted_for mutation. */ int raft_begin_pre_vote(raft_state_t *s); /* sends pre-votes, self-grant in raft_state_t */ int raft_on_pre_vote_request(raft_state_t *s, int from_idx, uint64_t cand_term, uint64_t cand_last_log_term, uint64_t cand_last_log_index); /* RAFT_GRANT=1 / RAFT_DENY=0 */ int raft_on_pre_vote_reply(raft_state_t *s, int peer_idx, int granted); /* tally in raft_state_t -> WON / LOST / PENDING */ ``` (Note: superseded by `raft-commit-safety` — `raft_on_pre_vote_request` gained the `cand_last_log_term`/`cand_last_log_index` params and an internal `raft_log_ok` freshness check; see the Responsibilities & Constraints note above.) - Preconditions: `raft_begin_pre_vote` requires `role == RAFT_FOLLOWER`. - Postconditions: on `RAFT_PRE_VOTE_WON` the caller invokes the unchanged `raft_start_election`. - Invariants: persistent state untouched (asserted in `test_raft.c`). **Implementation Notes** - Integration: `on_vote_eligible` is the existing `raft_cb_vote_eligible`; reusing it gives PRIMARY-deny + heartbeat-stickiness + catch-up-deny for free (Req 3.x). - Validation: unit tests assert `term`/`voted_for` unchanged after a granted PreVote. - Risks: accidental term mutation — mitigated by keeping all term writes in `raft_start_election` only. ### Controller — PreVote dispatch (cluster.c) | Field | Detail | |-------|--------| | Intent | Answer `preVoteRequest` with freshness + stickiness gates, no mutation; route `preVoteReply` to the tally | | Requirements | 2.1, 2.2, 3.1–3.3, 7.1, 7.2 | **Responsibilities & Constraints** - `OPCode_preVoteRequest`: parse `request_vote_req_t`; compute freshness floor via `ha_vote_self_freshness`; deny on freshness failure; otherwise call `raft_on_pre_vote_request`; reply `request_vote_reply_t` with **unchanged** `term`, the grant bit, and `peer_config_tag`/`peer_last_log_lsn`. Log denials with candidate id + reason (Req 7.2). - `OPCode_preVoteReply`: route to `pre_vote_reply_dispatch` → `raft_on_pre_vote_reply` (raft-library tally), paralleling the production `vote_reply_dispatch` → `raft_on_vote_reply`. - Inbound connections remain vote-only; the reply carries no state push. **Contracts**: Service [x] / Event [x] ##### Event Contract - Published: `preVoteRequest` (term+1, candidate_id, config_tag, last_log_lsn). - Subscribed: `preVoteReply` (term, vote_granted, peer_config_tag, peer_last_log_lsn). - Delivery: best-effort over the existing election transport; missing/denied replies count as non-grants. No ordering requirement beyond per-peer idempotency: `raft_on_pre_vote_reply` receives `peer_idx` and tracks a per-peer replied flag in `raft_state_t` to avoid double-counting duplicate replies. ### Controller — PreVote orchestration & trigger restructuring (ha.c) | Field | Detail | |-------|--------| | Intent | Run the side-effect-free round, gate the failover trigger, collect replies, branch to real election or clean abandon | | Requirements | 1.1, 1.4, 1.5, 4.1–4.4, 5.4, 7.3 | **Responsibilities & Constraints** - `ha_begin_failover`: single entry from both `ha_on_primary_ctrl_fd_rdhup` and `liveness_poll_fn`; starts `ha_start_prevote_async`; **does not** truncate WAL or transition to `CTRL_ELECTING` (deferred to PreVote success). - `ha_start_prevote_async`: call `raft_begin_pre_vote` (self-grant + send `preVoteRequest` per peer via `on_send_pre_vote_request`); arm an `election_timeout_ms` round timeout. The PreVote tally lives in `raft_state_t`, not a controller struct. - `pre_vote_reply_dispatch` → `raft_on_pre_vote_reply`: on `RAFT_PRE_VOTE_WON` → truncate + `CTRL_ELECTING` + `ha_start_election_async`; on `RAFT_PRE_VOTE_LOST` or round timeout → stay `CTRL_STANDBY`, re-arm liveness, log abandonment (Req 7.3). - `ha_on_heartbeat_received`: if a PreVote round is in progress, cancel it (clear the round handle) and re-attach as standby (Req 4.2) — no term change. - **Invariant**: while a PreVote round is in progress, `ha->role` stays `CTRL_STANDBY` and no WAL/term mutation occurs. **Contracts**: Service [x] / State [x] ##### State Management - State model: the PreVote tally (`pre_votes_granted`, `pre_vote_replies`, `pre_vote_in_progress`) lives in `raft_state_t`, mirroring the real-vote tally; quorum formula `(n_peers / 2) + 1` reused from `raft_on_vote_reply`. The controller holds only a round-timeout handle + in-progress flag for cancellation. - Concurrency: PreVote reply handling runs on the same single-threaded dispatch path as `vote_reply_dispatch`; no new lock introduced. **Implementation Notes** - Integration: PreVote success calls the **unchanged** `ha_start_election_async`/`raft_start_election`, so the real election (and bug-339/340 behavior) is unaffected. - Validation: `test_ha_prevote.c` asserts the frozen-rejoin no-op (term constant, primary not demoted) and genuine-failover success. - Risks: a failed PreVote leaving the watchdog un-armed (would freeze failover) — covered by an explicit re-arm test. ## Error Handling ### Error Strategy - **No PreVote majority** (expected, frozen-rejoin or split): not an error — abandon cleanly, stay `CTRL_STANDBY`, re-arm liveness, log at debug/info with reason. The cluster term is unchanged. - **Round timeout**: treated identically to "lost"; the next liveness cycle retries PreVote, succeeding once a majority's stickiness lapses (bounds legitimate failover by heartbeat-timeout, not unbounded — Req 5.4). - **Reply from a higher-term peer**: the PreVote reply path does **not** adopt the peer's term (PreVote is side-effect-free); a genuinely higher term is observed and adopted only through the normal real-vote / heartbeat paths, unchanged. - **Unknown candidate id / malformed payload**: deny PreVote (same as real-vote dispatch), log. ### Monitoring - Log every PreVote denial with candidate id + reason (`primary`, `fresh_heartbeat`, `catch_up`, `stale_term`, `freshness`) — Req 7.2. - Log campaign abandonment distinctly from genuine failover so operators can tell a suppressed disruptive election from a real one — Req 7.3. ## Testing Strategy ### Unit Tests (raft library — `test_raft.c`) - `raft_on_pre_vote_request` grants for a fresh, higher-term candidate when `on_vote_eligible` returns 1, and asserts `term`/`voted_for`/`role` unchanged (2.3, 6.2). - `raft_on_pre_vote_request` denies on `cand_term < term` and when `on_vote_eligible` returns 0 (2.2, 3.x). - `raft_on_pre_vote_reply` returns WON at quorum, LOST when all replied without quorum, with no role/term change (1.4, 1.5, 6.2). - A granted PreVote does not set `voted_for`, so a later real vote in the same term is still grantable (2.4). ### Integration Tests (controller — `test_ha_prevote.c`, `test_ha_vote_gate.c`) - **Frozen-then-resumed no-op**: 3-node, primary alive; a resumed standby starts PreVote; both peers deny (PRIMARY + fresh heartbeat); assert no real RequestVote sent, cluster term unchanged, primary still `CTRL_PRIMARY` (4.1, 4.3, 4.4). - **Genuine failover**: primary dead, peers' heartbeats stale; PreVote granted → real election → new primary within bounds (5.1, 5.4). - **Re-attach on heartbeat mid-round**: PreVote in progress, primary heartbeat arrives → round cancelled, node re-attaches as standby, no term change (4.2). - **PreVote eligibility reuse**: PRIMARY-deny, fresh-heartbeat-deny, catch-up-deny exercised through the PreVote path (3.1–3.3). - **Liveness re-arm after loss**: failed PreVote re-arms the watchdog; a subsequent genuine failure still triggers (1.4 robustness). ### Verification (Req 6) - `test_raft.c` invariant assertions and a `raft/raft.tla` consistency argument (RN-1, tracked in `research.md`) confirm Election Safety and term monotonicity hold with the PreVote path; the real-vote path is unchanged so bug-339/340 coverage is preserved (5.3).