commit missed

.kiro/specs/ha-prevote/design.md

@@ -0,0 +1,332 @@
+# Design Document
+
+## Overview
+
+**Purpose.** This feature adds a **PreVote phase** to the XCE controller HA election path so that a controller which is merely temporarily unreachable cannot inflate its persistent term or unseat a healthy primary. Before a node increments its term and campaigns, it must first win a hypothetical PreVote majority — a "would you vote for me in term+1?" round that mutates no node's persistent term or recorded vote.
+
+**Users.** Operators of 3-node controller HA clusters, and the controllers themselves, benefit: a frozen-then-resumed standby rejoins as a no-op instead of triggering the ~30s demote-and-restart re-election storm observed live on 2026-06-15.
+
+**Impact.** The election trigger path changes from `trigger → real election (term++)` to `trigger → PreVote round → (only on majority) → real election (term++)`. The real-vote state machine, replication, and commit engine are unchanged. The change is additive: two new wire opcodes reusing the existing vote payload structs, pure PreVote decision **and tally** functions in the raft library (mirroring the production real-vote path `raft_on_vote_reply`), and PreVote orchestration in the controller that reuses the existing eligibility gate.
+
+### Goals
+- A node never increments its persistent term unless a PreVote majority is first obtained.
+- A voter denies a PreVote whenever it still believes a leader is alive (leader stickiness), reusing the existing heartbeat-freshness predicate.
+- A frozen-then-resumed standby in a healthy cluster gets no PreVote majority and re-attaches as a standby with the cluster term unchanged.
+- Genuine leadership changes (true primary death, partition heal) still elect within existing timeout bounds.
+- All Raft safety invariants and the bug-339/bug-340 fixes are preserved.
+
+### Non-Goals
+- Changes to replication, the commit/barrier engine, snapshot/WAL catch-up transfer, or config bootstrap.
+- Redesign of the freshness / log-completeness comparison (reused as-is).
+- Backward/rolling-upgrade compatibility (product is pre-release; all controllers run the same build).
+- Leader-side CheckQuorum self-demotion and the defense-in-depth real-vote stickiness guard (see Architecture → Deferred decisions).
+- **The startup / first-boot election** (the `ha_elect_ctx_t.done` semaphore path). PreVote gates only the **failover** trigger; on first boot there is no leader to protect and every node's heartbeat is stale, so a PreVote round would trivially pass. The startup election path is untouched.
+
+## Boundary Commitments
+
+### This Spec Owns
+- The **PreVote round** on the election-trigger path: a side-effect-free majority probe that runs before any persistent term increment.
+- The **voter-side PreVote decision**: term check + reuse of the existing eligibility gate + reuse of the dispatcher freshness gate, with **zero persistent mutation**.
+- The two new wire opcodes (`preVoteRequest`, `preVoteReply`) and their dispatch.
+- The trigger-path restructuring so that `truncate-WAL + CTRL_ELECTING` happens only after PreVote succeeds.
+- PreVote observability (denial reasons, abandoned-campaign logging).
+
+### Out of Boundary
+- The real-vote state machine (`raft_on_vote_request`, `raft_on_vote_reply`, `raft_start_election`) — invoked unchanged after PreVote success.
+- The freshness floor computation (`ha_vote_self_freshness`) and the eligibility gate (`raft_cb_vote_eligible`) — reused, not modified (except that the eligibility gate is now also called from the PreVote receiver path).
+- Replication, commit barrier, snapshot transfer, demotion-convergence mechanics (truncate-above-committed + self-restart) — unchanged.
+- Leader self-demotion / CheckQuorum.
+- The startup-election `ha_elect_ctx_t.done` semaphore path — not gated by PreVote.
+
+### Allowed Dependencies
+- raft library callback seams (`raft_callbacks_t`), the raft-library vote-tally pattern (`raft_on_vote_reply` ⇒ `raft_on_pre_vote_reply`), `ha_heartbeat_is_fresh`, `ha_vote_self_freshness`, the capnp election transport (`xce_request` / vote endpoints / `peer_ctrl_msg_handler`), and the `xce_vote_request_hook` test seam.
+- Dependency direction (must not be violated): **capnp/types → raft library (pure) → controller adapter (ha.c) → dispatch/transport (cluster.c)**. The raft library must not depend on controller state.
+
+### Revalidation Triggers
+- Any change to the `request_vote_req_t` / `request_vote_reply_t` payload shape (shared by real-vote and PreVote).
+- Any change to `raft_cb_vote_eligible` semantics (now shared by both vote paths).
+- Any change to the leader-stickiness predicate (`ha_heartbeat_is_fresh` / `heartbeat_timeout_ms`).
+- Any change to the election-trigger entry points (`ha_on_primary_ctrl_fd_rdhup`, `liveness_poll_fn`).
+- Introduction of mixed-version clusters (would reintroduce the deferred wire-compat concern).
+
+## Architecture
+
+### Existing Architecture Analysis
+
+The election path is split across two layers (preserved by this design):
+
+- **raft library** (`src/lib/raft/raft.c`, `raft.h`) — the canonical, TLA+-checkable state machine. Owns term, `voted_for`, role, and the pure vote-decision logic. Communicates outward only through `raft_callbacks_t` (no controller dependency). Today: `raft_start_election` increments+persists the term before any RPC; `raft_on_vote_request` adopts a higher term and steps down a leader in **Step 2 — before** the `on_vote_eligible` gate runs. This ordering is exactly why the existing heartbeat-stickiness gate cannot protect a leader from a fresh, higher-term RequestVote, and why the fix must be sender-side.
+- **controller adapter** (`src/controller/ha.c`, `cluster.c`) — async transport + collection. `cluster.c` dispatches `OPCode_requestVote`, applies the freshness floor, maps candidate name→index, then calls the library. `ha.c` orchestrates `ha_start_election_async` (allocates `ha_elect_ctx_t`, self-vote, delegates to `raft_start_election`), collects replies (`handle_request_vote_reply` → `ha_elect_evaluate_and_signal`), and gates eligibility (`raft_cb_vote_eligible`: PRIMARY-deny, fresh-heartbeat-deny, catch-up-deny). Triggers: `ha_on_primary_ctrl_fd_rdhup` (RDHUP) and `liveness_poll_fn` (heartbeat timeout), which today transition to `CTRL_ELECTING`, truncate WAL, and start the election.
+
+### Architecture Pattern & Boundary Map
+
+```mermaid
+graph TB
+    subgraph trigger [Election Trigger]
+        Rdhup[ha_on_primary_ctrl_fd_rdhup]
+        Liveness[liveness_poll_fn]
+    end
+    subgraph controller [Controller Adapter ha.c cluster.c]
+        BeginFailover[ha_begin_failover]
+        PrevoteAsync[ha_start_prevote_async]
+        ElectAsync[ha_start_election_async]
+        Collect[raft_on_pre_vote_reply tally]
+        DispReq[cluster.c preVoteRequest case]
+        DispRep[cluster.c preVoteReply case]
+        Eligible[raft_cb_vote_eligible reused]
+        Fresh[ha_vote_self_freshness reused]
+    end
+    subgraph raftlib [Raft Library pure]
+        Begin[raft_begin_pre_vote]
+        OnReq[raft_on_pre_vote_request]
+        OnRep[raft_on_pre_vote_reply]
+        StartElection[raft_start_election unchanged]
+    end
+    subgraph wire [Transport]
+        Capnp[preVoteRequest preVoteReply opcodes]
+    end
+
+    Rdhup --> BeginFailover
+    Liveness --> BeginFailover
+    BeginFailover --> PrevoteAsync
+    PrevoteAsync --> Begin
+    Begin --> Capnp
+    Capnp --> DispReq
+    DispReq --> Fresh
+    DispReq --> Eligible
+    DispReq --> OnReq
+    OnReq --> Capnp
+    Capnp --> DispRep
+    DispRep --> OnRep
+    OnRep --> Collect
+    Collect -->|majority| ElectAsync
+    ElectAsync --> StartElection
+    Collect -->|lost or timeout| Rearm[re-arm liveness stay STANDBY]
+```
+
+**Architecture Integration**
+- **Selected pattern**: side-effect-free probe interposed before an existing state transition (Raft dissertation §9.6 PreVote). Adopted, not invented.
+- **Boundaries**: raft library owns the pure PreVote *decision and tally*; the controller owns the *orchestration/transport*. The voter eligibility gate is **shared** with the real-vote path via reuse, not duplication; the PreVote tally mirrors the production real-vote tally (`raft_on_vote_reply`) rather than the test-only `ha_elect_ctx_t` path.
+- **Existing patterns preserved**: callback-seam isolation of the raft library; raft-library-owns-tally (`raft_on_vote_reply` ⇒ `raft_on_pre_vote_reply`); dispatcher-applies-freshness-before-library; `xce_vote_request_hook` test seam.
+- **New components rationale**: a new opcode pair keeps the real-vote handler byte-for-byte unchanged (zero regression risk to bug-339/340); pure PreVote functions in the library keep the state machine TLA+-checkable.
+- **Steering compliance**: async, non-blocking vtask handlers; long-lived connections; inbound connections vote-only; immutable-term discipline (PreVote mutates nothing).
+
+### Deferred decisions (documented, not implemented)
+- **Defense-in-depth real-vote stickiness guard** — rejecting a real higher-term RequestVote while a leader is believed alive. PreVote already guarantees a candidate only sends a real RequestVote after a majority's stickiness has lapsed, so in a same-version cluster no stickiness-violating real RequestVote can arise. Adding the guard would touch the safety-critical real-vote term-adoption path and risk regressing bug-339/340 for no in-scope benefit. Deferred; revisit only if mixed-version clusters or leader CheckQuorum are introduced.
+
+### Technology Stack
+
+| Layer | Choice / Version | Role in Feature | Notes |
+|-------|------------------|-----------------|-------|
+| Messaging / Events | Cap'n Proto (capnpc-c 1.0.1) | New `preVoteRequest`/`preVoteReply` opcodes reusing existing vote payload structs | No new structs; follows existing opcode-enum pattern |
+| Backend / Services | C (raft library + controller) | Pure PreVote decision + tally functions; controller orchestration | No new libraries; reuses iomgr/delaymgr and the raft tally pattern |
+| Infrastructure / Runtime | iomgr async transport, delaymgr | PreVote send/collect + round timeout | Non-blocking, mirrors election path |
+
+## File Structure Plan
+
+### Modified Files
+- `src/schema/controller.capnp` — add two `OPCode` enum entries: `preVoteRequest`, `preVoteReply`. Reuse existing `RequestVoteRequest` / `RequestVoteReply` structs and their `Request.body` union arms (`requestvotemsg` / `requestvotereplymsg`); no new payload structs.
+- `src/lib/raft/raft.h` — add pre-vote return codes (e.g. `RAFT_PRE_VOTE_WON`, `RAFT_PRE_VOTE_LOST`, `RAFT_PRE_VOTE_PENDING`), an optional `on_send_pre_vote_request` callback in `raft_callbacks_t`, pre-vote tally fields in `raft_state_t` (`pre_votes_granted`, `pre_vote_replies`, `pre_vote_in_progress`), and declarations for the three new functions.
+- `src/lib/raft/raft.c` — add `raft_begin_pre_vote`, `raft_on_pre_vote_request`, `raft_on_pre_vote_reply` (all pure: no `on_persist`, no `term`/`voted_for`/`role` mutation). `raft_on_pre_vote_request` reuses `on_vote_eligible`. **`raft_on_pre_vote_reply` owns the PreVote tally in `raft_state_t` (mirroring `raft_on_vote_reply`) and returns `WON/LOST/PENDING` — it never fires `on_become_leader`/`on_become_follower`.**
+- `src/controller/ha.h` — add a controller-side PreVote round handle on `ha_state_t` (round-timeout context + in-progress flag for cancellation). **No change to `ha_elect_ctx_t`** (the tally lives in `raft_state_t`, not here).
+- `src/controller/ha.c` — add `ha_begin_failover` (common entry), `ha_start_prevote_async` (calls `raft_begin_pre_vote`, arms round timeout), and `pre_vote_reply_dispatch` (calls `raft_on_pre_vote_reply`, mirroring `vote_reply_dispatch → raft_on_vote_reply`); on `RAFT_PRE_VOTE_WON` perform truncate + `CTRL_ELECTING` + `ha_start_election_async`; on `RAFT_PRE_VOTE_LOST`/round-timeout stay `CTRL_STANDBY`, re-arm liveness, log abandonment. Modify `ha_on_primary_ctrl_fd_rdhup` and `liveness_poll_fn` to call `ha_begin_failover` (PreVote first; **no** truncate/`CTRL_ELECTING` until PreVote wins); modify `ha_on_heartbeat_received` to cancel an in-progress PreVote and re-attach.
+- `src/controller/cluster.c` — add `OPCode_preVoteRequest` and `OPCode_preVoteReply` dispatch cases mirroring the real-vote cases, calling `raft_on_pre_vote_request` (no term adoption) and routing replies to `pre_vote_reply_dispatch`.
+
+### New Files
+- `src/controller/testsuite/test_ha_prevote.c` — PreVote gating + frozen-then-resumed no-op scenario + genuine-failover scenario.
+
+### Test files extended
+- `src/lib/testsuite/test_raft.c` — pure PreVote decision/tally unit tests + invariant assertions (no term mutation).
+- `src/controller/testsuite/test_ha_vote_gate.c` — PreVote eligibility reuse of the role/heartbeat/catch-up gates.
+
+## System Flows
+
+### PreVote round (candidate side)
+
+```mermaid
+sequenceDiagram
+    participant Trig as Trigger
+    participant Cand as Candidate ha.c
+    participant Peer as Peer voter
+    Trig->>Cand: ha_begin_failover (stay CTRL_STANDBY)
+    Cand->>Cand: ha_start_prevote_async (prospective term = current+1, no persist)
+    Cand->>Peer: preVoteRequest term+1 tag lsn
+    Note over Peer: stickiness gate then freshness gate then term check
+    Peer-->>Cand: preVoteReply granted or denied (peer term unchanged)
+    alt PreVote majority
+        Cand->>Cand: now truncate WAL, CTRL_ELECTING
+        Cand->>Cand: ha_start_election_async then raft_start_election (term++)
+    else lost or round timeout
+        Cand->>Cand: stay CTRL_STANDBY, re-arm liveness watchdog
+        Note over Cand: next primary heartbeat re-attaches as standby
+    end
+```
+
+Key decisions: the candidate remains `CTRL_STANDBY` and mutates nothing during the PreVote round; the destructive `truncate + CTRL_ELECTING` step is deferred to PreVote success. An incoming primary heartbeat during the round cancels it and re-attaches. The round is bounded by `election_timeout_ms`; on loss the watchdog is re-armed so a later genuine failure still triggers, and repeated PreVote attempts succeed once a majority's stickiness lapses.
+
+### PreVote decision (voter side)
+
+```mermaid
+flowchart TD
+    A[preVoteRequest received] --> B{role == CTRL_PRIMARY}
+    B -- yes --> D[deny]
+    B -- no --> C{role == CTRL_STANDBY and heartbeat fresh}
+    C -- yes --> D
+    C -- no --> E{in catch-up}
+    E -- yes --> D
+    E -- no --> F{req.term lt current_term}
+    F -- yes --> D
+    F -- no --> G{freshness floor ok}
+    G -- no --> D
+    G -- yes --> H[grant, no term change, no persist, no voted_for]
+```
+
+Steps B/C/E are exactly the existing `raft_cb_vote_eligible` gates (reused). Step G is exactly the existing dispatcher freshness floor (`ha_vote_self_freshness`). The only PreVote-specific rule is the **absence of any state mutation** on either grant or deny.
+
+## Requirements Traceability
+
+| Requirement | Summary | Components | Interfaces / Flows |
+|-------------|---------|------------|--------------------|
+| 1.1 | PreVote before term increment on trigger | `ha_begin_failover`, `ha_start_prevote_async` | PreVote round flow |
+| 1.2 | No term/vote mutation during round | `raft_begin_pre_vote`, `ha_start_prevote_async` | candidate stays STANDBY |
+| 1.3 | Request prospective term+1, not persisted | `raft_begin_pre_vote` | preVoteRequest payload |
+| 1.4 | No majority → no real election | `raft_on_pre_vote_reply` (LOST), `pre_vote_reply_dispatch` | round flow "lost" path |
+| 1.5 | Majority → start real election | `ha_start_election_async`, `raft_start_election` (unchanged) | round flow "won" path |
+| 2.1 | Grant only if real-vote freshness gate passes | `cluster.c preVoteRequest`, `ha_vote_self_freshness` | voter flow step G |
+| 2.2 | Fail freshness → deny | same | voter flow step G |
+| 2.3 | Grant/deny records no vote, no term | `raft_on_pre_vote_request` | voter flow leaf H |
+| 2.4 | PreVote does not consume the real vote | `raft_on_pre_vote_request` (no `voted_for` write) | — |
+| 3.1 | Deny while leader believed alive | `raft_cb_vote_eligible` (`ha_heartbeat_is_fresh`) | voter flow step C |
+| 3.2 | Past self-election threshold → freshness-only eval | `raft_cb_vote_eligible` | voter flow C→E→F→G |
+| 3.3 | Primary denies PreVotes | `raft_cb_vote_eligible` (PRIMARY-deny) | voter flow step B |
+| 4.1 | Frozen-resumed gets no majority | full path | frozen-rejoin scenario test |
+| 4.2 | Re-attach on next heartbeat, no term change | `ha_on_heartbeat_received` | round flow "lost"/cancel |
+| 4.3 | Healthy primary not demoted/restarted | PreVote denies → no real RequestVote | — |
+| 4.4 | Term unchanged, no re-election cycle | candidate stays STANDBY | — |
+| 5.1 | True death → PreVotes granted → elect | `raft_cb_vote_eligible` (stale heartbeat → eligible) | genuine-failover test |
+| 5.2 | Partition heal → adopt higher term | `raft_on_vote_reply` (unchanged) | real-vote path |
+| 5.3 | No regression of bug-339/bug-340 | real-vote path untouched | — |
+| 5.4 | Complete within election-timeout bounds | `election_timeout_ms` round timeout | round flow |
+| 6.1 | Election Safety preserved | real-vote path untouched; PreVote grants no leadership | RN-1 (research.md) |
+| 6.2 | Term monotonicity; PreVote never changes term | `raft_on_pre_vote_*` (pure) | — |
+| 6.3 | Leader Completeness, monotonic commit index | freshness floor reused; commit path untouched | — |
+| 6.4 | Consistent with raft.tla + checklist | `test_raft.c` invariant tests | RN-1 |
+| 7.1 | PreVote over existing transport, no disruption to real votes | new opcodes, reused arms | — |
+| 7.2 | Log PreVote denials with candidate + reason | `cluster.c preVoteRequest`, `on_trace`/`tlog` | — |
+| 7.3 | Log abandoned campaign | `pre_vote_reply_dispatch` (LOST/timeout) | round flow |
+
+## Components and Interfaces
+
+| Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
+|-----------|-------|--------|--------------|--------------------------|-----------|
+| raft pre-vote functions | raft library | Pure PreVote decision + tally | 1.2, 1.3, 2.3, 2.4, 6.2 | `on_vote_eligible` (P0), `on_send_pre_vote_request` (P1) | Service, State |
+| PreVote dispatch | cluster.c | Receive/answer PreVote, no mutation | 2.1, 2.2, 3.1–3.3, 7.1, 7.2 | `ha_vote_self_freshness` (P0), `raft_on_pre_vote_request` (P0) | Service, Event |
+| PreVote orchestration | ha.c | Run round, gate trigger, branch to election | 1.1, 1.4, 1.5, 4.x, 5.4, 7.3 | `raft_on_pre_vote_reply` (P0), `ha_start_election_async` (P0), `liveness` (P1) | Service, State |
+| Trigger restructuring | ha.c | Defer truncate/CTRL_ELECTING until PreVote wins | 1.1, 4.2, 4.3, 4.4 | `ha_on_primary_ctrl_fd_rdhup`, `liveness_poll_fn` (P0) | State |
+
+### Raft library — pre-vote functions
+
+| Field | Detail |
+|-------|--------|
+| Intent | Pure PreVote decision and tally; canonical, TLA+-checkable; mutates no persistent state |
+| Requirements | 1.2, 1.3, 2.3, 2.4, 6.2 |
+
+**Responsibilities & Constraints**
+- `raft_begin_pre_vote`: compute prospective term = `term + 1` **without** assigning it to `s->term` or persisting; set `pre_vote_in_progress`, `pre_votes_granted = 1` (self); emit `on_send_pre_vote_request(ctx, peer_idx, prospective_term)` per peer.
+- `raft_on_pre_vote_request`: deny if `cand_term < s->term`; else return the result of `on_vote_eligible(ctx, from_idx, cand_term)`; **never** touches `term`, `voted_for`, `role`, or calls `on_persist`. Freshness is applied by the dispatcher before this call (same contract as real vote).
+- `raft_on_pre_vote_reply`: **owns the PreVote tally** in `raft_state_t` (`pre_votes_granted`, `pre_vote_replies`), mirroring `raft_on_vote_reply`; return `RAFT_PRE_VOTE_WON` at quorum, `RAFT_PRE_VOTE_LOST` when all replied without quorum, else `RAFT_PRE_VOTE_PENDING`. Unlike `raft_on_vote_reply`, it **never** fires `on_become_leader`/`on_become_follower` and makes no role/term change — the controller acts on the returned status.
+- **Invariant**: across all three functions, `s->term`, `s->voted_for`, `s->voted_for_term`, and `s->role` are unchanged, and `on_persist` is never invoked.
+
+**Contracts**: Service [x] / State [x]
+
+##### Service Interface
+```c
+/* All pure: no persist, no term/role/voted_for mutation. */
+int raft_begin_pre_vote(raft_state_t *s);                 /* sends pre-votes, self-grant in raft_state_t */
+int raft_on_pre_vote_request(raft_state_t *s,
+                             int from_idx, uint64_t cand_term);  /* 1=grant, 0=deny */
+int raft_on_pre_vote_reply(raft_state_t *s,
+                           int peer_idx, int granted);    /* tally in raft_state_t -> WON / LOST / PENDING */
+```
+- Preconditions: `raft_begin_pre_vote` requires `role == RAFT_FOLLOWER`.
+- Postconditions: on `RAFT_PRE_VOTE_WON` the caller invokes the unchanged `raft_start_election`.
+- Invariants: persistent state untouched (asserted in `test_raft.c`).
+
+**Implementation Notes**
+- Integration: `on_vote_eligible` is the existing `raft_cb_vote_eligible`; reusing it gives PRIMARY-deny + heartbeat-stickiness + catch-up-deny for free (Req 3.x).
+- Validation: unit tests assert `term`/`voted_for` unchanged after a granted PreVote.
+- Risks: accidental term mutation — mitigated by keeping all term writes in `raft_start_election` only.
+
+### Controller — PreVote dispatch (cluster.c)
+
+| Field | Detail |
+|-------|--------|
+| Intent | Answer `preVoteRequest` with freshness + stickiness gates, no mutation; route `preVoteReply` to the tally |
+| Requirements | 2.1, 2.2, 3.1–3.3, 7.1, 7.2 |
+
+**Responsibilities & Constraints**
+- `OPCode_preVoteRequest`: parse `request_vote_req_t`; compute freshness floor via `ha_vote_self_freshness`; deny on freshness failure; otherwise call `raft_on_pre_vote_request`; reply `request_vote_reply_t` with **unchanged** `term`, the grant bit, and `peer_config_tag`/`peer_last_log_lsn`. Log denials with candidate id + reason (Req 7.2).
+- `OPCode_preVoteReply`: route to `pre_vote_reply_dispatch` → `raft_on_pre_vote_reply` (raft-library tally), paralleling the production `vote_reply_dispatch` → `raft_on_vote_reply`.
+- Inbound connections remain vote-only; the reply carries no state push.
+
+**Contracts**: Service [x] / Event [x]
+
+##### Event Contract
+- Published: `preVoteRequest` (term+1, candidate_id, config_tag, last_log_lsn).
+- Subscribed: `preVoteReply` (term, vote_granted, peer_config_tag, peer_last_log_lsn).
+- Delivery: best-effort over the existing election transport; missing/denied replies count as non-grants. No ordering requirement beyond per-peer idempotency: `raft_on_pre_vote_reply` receives `peer_idx` and tracks a per-peer replied flag in `raft_state_t` to avoid double-counting duplicate replies.
+
+### Controller — PreVote orchestration & trigger restructuring (ha.c)
+
+| Field | Detail |
+|-------|--------|
+| Intent | Run the side-effect-free round, gate the failover trigger, collect replies, branch to real election or clean abandon |
+| Requirements | 1.1, 1.4, 1.5, 4.1–4.4, 5.4, 7.3 |
+
+**Responsibilities & Constraints**
+- `ha_begin_failover`: single entry from both `ha_on_primary_ctrl_fd_rdhup` and `liveness_poll_fn`; starts `ha_start_prevote_async`; **does not** truncate WAL or transition to `CTRL_ELECTING` (deferred to PreVote success).
+- `ha_start_prevote_async`: call `raft_begin_pre_vote` (self-grant + send `preVoteRequest` per peer via `on_send_pre_vote_request`); arm an `election_timeout_ms` round timeout. The PreVote tally lives in `raft_state_t`, not a controller struct.
+- `pre_vote_reply_dispatch` → `raft_on_pre_vote_reply`: on `RAFT_PRE_VOTE_WON` → truncate + `CTRL_ELECTING` + `ha_start_election_async`; on `RAFT_PRE_VOTE_LOST` or round timeout → stay `CTRL_STANDBY`, re-arm liveness, log abandonment (Req 7.3).
+- `ha_on_heartbeat_received`: if a PreVote round is in progress, cancel it (clear the round handle) and re-attach as standby (Req 4.2) — no term change.
+- **Invariant**: while a PreVote round is in progress, `ha->role` stays `CTRL_STANDBY` and no WAL/term mutation occurs.
+
+**Contracts**: Service [x] / State [x]
+
+##### State Management
+- State model: the PreVote tally (`pre_votes_granted`, `pre_vote_replies`, `pre_vote_in_progress`) lives in `raft_state_t`, mirroring the real-vote tally; quorum formula `(n_peers / 2) + 1` reused from `raft_on_vote_reply`. The controller holds only a round-timeout handle + in-progress flag for cancellation.
+- Concurrency: PreVote reply handling runs on the same single-threaded dispatch path as `vote_reply_dispatch`; no new lock introduced.
+
+**Implementation Notes**
+- Integration: PreVote success calls the **unchanged** `ha_start_election_async`/`raft_start_election`, so the real election (and bug-339/340 behavior) is unaffected.
+- Validation: `test_ha_prevote.c` asserts the frozen-rejoin no-op (term constant, primary not demoted) and genuine-failover success.
+- Risks: a failed PreVote leaving the watchdog un-armed (would freeze failover) — covered by an explicit re-arm test.
+
+## Error Handling
+
+### Error Strategy
+- **No PreVote majority** (expected, frozen-rejoin or split): not an error — abandon cleanly, stay `CTRL_STANDBY`, re-arm liveness, log at debug/info with reason. The cluster term is unchanged.
+- **Round timeout**: treated identically to "lost"; the next liveness cycle retries PreVote, succeeding once a majority's stickiness lapses (bounds legitimate failover by heartbeat-timeout, not unbounded — Req 5.4).
+- **Reply from a higher-term peer**: the PreVote reply path does **not** adopt the peer's term (PreVote is side-effect-free); a genuinely higher term is observed and adopted only through the normal real-vote / heartbeat paths, unchanged.
+- **Unknown candidate id / malformed payload**: deny PreVote (same as real-vote dispatch), log.
+
+### Monitoring
+- Log every PreVote denial with candidate id + reason (`primary`, `fresh_heartbeat`, `catch_up`, `stale_term`, `freshness`) — Req 7.2.
+- Log campaign abandonment distinctly from genuine failover so operators can tell a suppressed disruptive election from a real one — Req 7.3.
+
+## Testing Strategy
+
+### Unit Tests (raft library — `test_raft.c`)
+- `raft_on_pre_vote_request` grants for a fresh, higher-term candidate when `on_vote_eligible` returns 1, and asserts `term`/`voted_for`/`role` unchanged (2.3, 6.2).
+- `raft_on_pre_vote_request` denies on `cand_term < term` and when `on_vote_eligible` returns 0 (2.2, 3.x).
+- `raft_on_pre_vote_reply` returns WON at quorum, LOST when all replied without quorum, with no role/term change (1.4, 1.5, 6.2).
+- A granted PreVote does not set `voted_for`, so a later real vote in the same term is still grantable (2.4).
+
+### Integration Tests (controller — `test_ha_prevote.c`, `test_ha_vote_gate.c`)
+- **Frozen-then-resumed no-op**: 3-node, primary alive; a resumed standby starts PreVote; both peers deny (PRIMARY + fresh heartbeat); assert no real RequestVote sent, cluster term unchanged, primary still `CTRL_PRIMARY` (4.1, 4.3, 4.4).
+- **Genuine failover**: primary dead, peers' heartbeats stale; PreVote granted → real election → new primary within bounds (5.1, 5.4).
+- **Re-attach on heartbeat mid-round**: PreVote in progress, primary heartbeat arrives → round cancelled, node re-attaches as standby, no term change (4.2).
+- **PreVote eligibility reuse**: PRIMARY-deny, fresh-heartbeat-deny, catch-up-deny exercised through the PreVote path (3.1–3.3).
+- **Liveness re-arm after loss**: failed PreVote re-arms the watchdog; a subsequent genuine failure still triggers (1.4 robustness).
+
+### Verification (Req 6)
+- `test_raft.c` invariant assertions and a `raft/raft.tla` consistency argument (RN-1, tracked in `research.md`) confirm Election Safety and term monotonicity hold with the PreVote path; the real-vote path is unchanged so bug-339/340 coverage is preserved (5.3).

.kiro/specs/ha-prevote/requirements.md

@@ -0,0 +1,211 @@
+# Requirements Document
+
+## Project Description (Input)
+
+**Who has the problem.** Operators of XCE 3-node controller HA clusters, and the
+controllers themselves, when a controller is temporarily unreachable to its
+peers but not crashed — e.g. a process freeze (SIGSTOP, long GC/allocator
+stall), a one-way or transient network partition, or a slow host — and then
+becomes reachable again ("frozen-then-resumed" / "rejoining" node).
+
+**Current situation.** The XCE election layer (Raft library `src/lib/raft/raft.c`
+plus the controller adapter in `src/controller/cluster.c`, `ha.c`, `myapp.c`)
+has no PreVote phase. A node that misses heartbeats hits its liveness timeout
+(`liveness_poll_fn` / RDHUP → `ha_on_primary_ctrl_fd_rdhup`) and immediately
+starts a real election, which **increments its persistent term** before any RPC
+(`raft_start_election`, `raft.c:68`). When such a node rejoins with an inflated
+term and sends `RequestVote`, the live primary adopts the higher term and steps
+down: `raft_on_vote_request` (`raft.c:125-134`, Step 2) updates `term`, sets
+role to follower, and fires `on_become_follower` **before** any freshness or
+role gate. The controller dispatcher (`cluster.c` `OPCode_requestVote`) only
+guards this with the `freshness_ok` check, so a rejoining node that was
+caught-up before the stall is fresh enough to pass and topples the leader.
+Because `raft_cb_become_follower` treats a was-PRIMARY demotion as
+demotion-convergence (truncate-above-committed-LSN + **self-restart**,
+`ha.c`), a single disruptive `RequestVote` makes the healthy primary truncate
+and restart, triggering a multi-term re-election storm.
+
+Observed live (2026-06-15, 3-node ASAN deploy): a SIGSTOP'd-then-resumed standby
+forced the primary to demote and self-restart; terms climbed 3 → 6 over ~30s
+before the cluster reconverged. No committed data was lost (`committed_lsn` and
+all submitted jobs survived), and the cluster self-healed — but availability
+blipped for ~30s on an event that should be a no-op. A normal crash+restart does
+NOT trigger this: a restarted node restores its term from disk and observes
+`leader already known → skipping own election` (`myapp.c`). The problem is
+specific to nodes that inflate their term without a genuine leadership change.
+
+This is the classic Raft "disruptive server" problem (Raft dissertation §9.6).
+The repository's TLA+ reference (`raft/raft.tla`) and the controller CLAUDE.md
+verification checklist require term monotonicity and Election Safety to be
+preserved by any change to this path.
+
+**What should change.** Introduce a PreVote phase so a node cannot inflate its
+term or disrupt a healthy leader merely by becoming temporarily unreachable.
+Before incrementing its term and campaigning, a candidate must first win a
+PreVote majority — a hypothetical "would you vote for me in term+1?" round that
+does NOT change any node's persistent term or recorded vote. Peers grant a
+PreVote only if they (a) would grant the corresponding real vote under the
+existing freshness/log-completeness gate AND (b) have not heard from a current
+leader within the minimum election timeout (leader stickiness). A node that has
+a live primary connection (fresh heartbeat / valid `primary_ctrl_fd`) denies
+PreVotes, so a rejoining node in a cluster with a healthy leader gets no PreVote
+majority, never increments its term, and simply re-attaches as a standby when
+the primary's next heartbeat/ctrlhello arrives.
+
+The change must:
+- Preserve all existing Raft safety invariants (Election Safety ≤1 leader/term,
+  Leader Completeness, term monotonicity, monotonic commit index) and remain
+  consistent with `raft/raft.tla` and the controller verification checklist.
+- Not regress the already-fixed failover (bug-339) and rejoin (bug-340) paths,
+  nor the legitimate cases where a real leadership change SHOULD occur (true
+  primary death, network healing into a higher-term majority).
+- Integrate with the existing two-tier transport (capnp `RequestVote` /
+  `RequestVoteReply` over iomgr, the `ha_elect_ctx_t` vote collection, and the
+  raft library callback seams) without breaking the unit-test seams.
+
+Scope: the raft library election state machine and its PreVote RPC, the
+controller vote dispatch/freshness/heartbeat-stickiness gates, the election
+trigger path (liveness timeout / RDHUP → PreVote → real election), and the
+wire message (a PreVote flag on the existing vote RPC or a new PreVote message).
+Out of scope: changes to replication, the commit engine, snapshot transfer, or
+config bootstrap beyond what the PreVote gating requires.
+
+## Introduction
+
+This feature adds a **PreVote phase** to the XCE controller HA election path so
+that a controller which is merely temporarily unreachable cannot inflate its
+term or unseat a healthy primary. Before a node increments its persistent term
+and starts a real election, it must first win a hypothetical PreVote majority.
+Peers grant a PreVote only when the candidate could legitimately become leader
+(the existing freshness / log-completeness gate) **and** the peer no longer
+believes a current leader is alive (leader stickiness). The net effect: a
+frozen-then-resumed standby in a cluster with a healthy primary gets no PreVote
+majority, never increments its term, and re-attaches as a standby — while
+genuine leadership changes (true primary death, healing into a higher-term
+majority) still proceed. The change must preserve every existing Raft safety
+invariant and must not regress the previously fixed failover (bug-339) and
+rejoin (bug-340) behavior.
+
+Throughout this document, "the Election Module" refers to the controller's HA
+election state machine (raft library plus its controller adapter) as observed
+through cluster behavior; "the voting controller" refers to a peer evaluating an
+incoming PreVote request; and "the HA cluster" refers to the externally observed
+behavior of the 3-node controller cluster as a whole.
+
+## Boundary Context
+
+- **In scope**:
+  - A PreVote round that precedes any persistent term increment and real
+    election on the election-trigger path (liveness timeout / primary-connection
+    loss).
+  - Voter-side PreVote evaluation combining the existing freshness /
+    log-completeness gate with a new leader-stickiness gate.
+  - The user-/operator-observable outcomes for frozen-then-resumed rejoin,
+    genuine failover, and partition healing.
+  - PreVote requests and replies traveling over the existing election transport.
+- **Out of scope**:
+  - Replication, the commit/barrier engine, snapshot/WAL catch-up transfer, and
+    config bootstrap — except where PreVote gating must read state those paths
+    already expose.
+  - Redesign of the existing freshness / log-completeness comparison itself; this
+    feature reuses it as the real-vote eligibility criterion.
+- **Adjacent expectations** (relied on, not owned by this feature):
+  - The existing freshness / log-completeness gate (config-tag and last-log-LSN
+    floor) remains the authority on whether a candidate could win a real vote.
+  - The existing heartbeat-liveness / primary-connection tracking remains the
+    signal for whether a peer still believes a leader is alive.
+  - The existing two-tier election transport and vote-collection mechanism
+    continue to deliver and tally PreVote and real-vote messages.
+  - Term restore-from-disk on crash+restart continues to suppress unnecessary
+    elections; this feature does not change that path.
+
+## Requirements
+
+### Requirement 1: PreVote precedes any persistent term increment
+
+**Objective:** As an XCE controller node, I want to win a hypothetical PreVote
+majority before starting a real election, so that I never inflate my persistent
+term unless a genuine leadership change is actually possible.
+
+#### Acceptance Criteria
+
+1. When a node's election trigger fires (liveness/heartbeat timeout or primary-connection RDHUP), the Election Module shall initiate a PreVote round before incrementing its persistent term.
+2. While a PreVote round is in progress, the Election Module shall not modify the node's persistent term or recorded vote.
+3. When a node requests PreVotes, the Election Module shall request them for the prospective next term (current term + 1) without persisting that prospective term.
+4. If a node does not obtain a PreVote majority, then the Election Module shall not start a real election and shall not increment its persistent term.
+5. When a node obtains a PreVote majority, the Election Module shall start a real election by incrementing its persistent term and casting its self-vote, following the existing real-election path.
+
+### Requirement 2: Voter-side PreVote eligibility reuses the real-vote gate
+
+**Objective:** As a voting controller, I want to grant a PreVote only to a
+candidate I would actually elect, so that PreVote screens out candidates that
+could never legitimately become leader.
+
+#### Acceptance Criteria
+
+1. When a voting controller receives a PreVote request, the voting controller shall grant it only if the candidate would also pass the existing real-vote freshness and log-completeness gate (config-tag and last-log-LSN floor).
+2. If a PreVote candidate fails the freshness or log-completeness gate, then the voting controller shall deny the PreVote.
+3. When a voting controller grants or denies a PreVote, the voting controller shall not record a vote, shall not change its voted-for state, and shall not change its persistent term.
+4. When a voting controller has granted a PreVote to one candidate, the voting controller shall remain able to grant a real vote later in the same term to any eligible candidate, regardless of which candidate received the PreVote.
+
+### Requirement 3: Leader stickiness gate
+
+**Objective:** As a voting controller that still believes its leader is alive, I
+want to deny PreVotes, so that a temporarily-unreachable peer cannot disrupt a
+healthy primary.
+
+#### Acceptance Criteria
+
+1. While a voting controller still considers a current leader alive — it has received leader traffic recently enough that it would not itself start an election — the voting controller shall deny all PreVote requests, even when the candidate passes the freshness gate.
+2. If a voting controller has gone long enough without leader traffic that it would itself be eligible to start an election, then the voting controller shall evaluate PreVote requests under the freshness / log-completeness gate alone.
+3. While a controller is itself the current primary holding quorum, the controller shall deny all PreVote requests.
+
+### Requirement 4: Frozen-then-resumed rejoin is non-disruptive
+
+**Objective:** As an operator, I want a frozen-then-resumed standby to rejoin
+without disrupting the cluster, so that transient unreachability is effectively a
+no-op for availability.
+
+#### Acceptance Criteria
+
+1. When a previously-frozen standby resumes and initiates a PreVote round while a healthy primary still holds the cluster, the HA cluster shall deny it a PreVote majority so that it neither increments its term nor sends a real RequestVote.
+2. When a non-disruptive rejoining standby next receives a primary heartbeat or ctrlhello, the standby shall re-attach as a standby under the existing primary without any intervening persistent-term change.
+3. While a healthy primary holds quorum, the HA cluster shall not demote, truncate-and-converge, or self-restart the primary in response to a rejoining standby's election attempt.
+4. When a frozen-then-resumed event occurs in an otherwise healthy 3-node cluster, the HA cluster shall keep the cluster's persistent term unchanged across the event and shall not enter a multi-term re-election cycle.
+
+### Requirement 5: Legitimate leadership changes still proceed
+
+**Objective:** As an XCE HA cluster, I want genuine leadership changes to
+complete, so that PreVote suppresses only disruptive elections and never blocks
+real failover.
+
+#### Acceptance Criteria
+
+1. When the primary genuinely fails and no peer has heard from a leader within its election-eligibility window, the HA cluster shall grant PreVotes to an eligible candidate so that a real election proceeds and a new primary is elected.
+2. When a partition heals and a node observes a higher-term majority, the Election Module shall still adopt the higher term and step down according to the existing term-monotonicity rules.
+3. The Election Module shall preserve the previously fixed failover behavior (bug-339) and rejoin behavior (bug-340) with PreVote enabled.
+4. Where a real leadership change is warranted, the HA cluster shall complete the new election within the existing election-timeout bounds, introducing no additional unbounded delay from the PreVote round.
+
+### Requirement 6: Raft safety invariants preserved
+
+**Objective:** As an XCE HA cluster, I want every existing Raft safety invariant
+preserved, so that adding PreVote introduces no correctness regression.
+
+#### Acceptance Criteria
+
+1. The Election Module shall preserve Election Safety (at most one leader per term) across all PreVote and real-vote paths.
+2. The Election Module shall preserve term monotonicity, such that no PreVote causes any node's persistent term to decrease and no node's persistent term increases without a completed real election.
+3. The HA cluster shall preserve Leader Completeness and a monotonic commit index across PreVote-gated elections.
+4. The PreVote behavior shall remain consistent with the repository's TLA+ reference (`raft/raft.tla`) and the controller verification checklist for term monotonicity and Election Safety.
+
+### Requirement 7: PreVote transport compatibility and operator observability
+
+**Objective:** As an operator, I want PreVote activity to be diagnosable and to
+not break existing election exchanges, so that I can distinguish a suppressed
+disruptive election from a real failover.
+
+#### Acceptance Criteria
+
+1. The HA cluster shall carry PreVote requests and replies over the existing election transport without disrupting real RequestVote / RequestVoteReply exchanges.
+2. When a controller denies a PreVote due to leader stickiness or the freshness gate, the HA cluster shall record the denial with the candidate identity and the denial reason at a diagnosable log level.
+3. When a node abandons a campaign after failing to obtain a PreVote majority, the HA cluster shall record that outcome so operators can distinguish a suppressed disruptive election from a genuine failover.

.kiro/specs/ha-prevote/spec.json

@@ -0,0 +1,22 @@
+{
+  "feature_name": "ha-prevote",
+  "created_at": "2026-06-15T23:45:43Z",
+  "updated_at": "2026-06-16T00:55:00Z",
+  "language": "en",
+  "phase": "tasks-generated",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": true
+    },
+    "design": {
+      "generated": true,
+      "approved": true
+    },
+    "tasks": {
+      "generated": true,
+      "approved": true
+    }
+  },
+  "ready_for_implementation": true
+}

.kiro/steering/product.md

@@ -21,7 +21,7 @@ XCE is an HPC (High-Performance Computing) cluster management system providing j
 
 XCE provides a **fully integrated single-vendor stack** (no database setup, unified build system) with an open, buildable codebase (Nix flakes, AppImage packaging). Compared to Slurm/PBS, it eliminates the external database requirement. Compared to commercial LSF, it is open source and reproducibly buildable.
 
-**Key architectural trade-off**: single-writer model throughout — only the primary writes. HA follows Raft semantics with CP guarantees: consensus election (RequestVote, term-based voting, freshness gates) via `raft.c`/`raft.h`, and majority-ACK log commitment for state-changing writes (WAL entries ≈ Raft log, `committed_lsn` ≈ commitIndex). Operations that cannot reach majority are rejected, never locally degraded. The single-writer constraint is XCE's simplification of Raft — no concurrent client proposals — while commitment and election inherit Raft's safety rules. All controller state is in-memory; persistence is WAL + checkpoint, not a traditional database.
+**Key architectural trade-off**: single-writer model throughout — only the primary writes. HA follows Raft semantics with CP guarantees: consensus election (RequestVote, term-based voting, freshness gates) via the `src/lib/raft/` shared module (`raft.c`/`raft.h`, consumed by the controller through `ha_init_raft`), and majority-ACK log commitment for state-changing writes (WAL entries ≈ Raft log, `committed_lsn` ≈ commitIndex). Operations that cannot reach majority are rejected, never locally degraded. The single-writer constraint is XCE's simplification of Raft — no concurrent client proposals — while commitment and election inherit Raft's safety rules. All controller state is in-memory; persistence is WAL + checkpoint, not a traditional database.
 
 ## Active Development Context
 
@@ -39,10 +39,13 @@ The project is in a **persistence and HA transformation** (branch `persistent_ha
 
 Features are designed and implemented via the kiro spec system (`.kiro/specs/`).
 Each feature follows a 3-phase approval workflow: Requirements → Design → Tasks → Implementation.
-Active specs (10 total) include: ha-config-bootstrap, ha-heartbeat-liveness,
+Active specs (15 total). The HA consistency family is the current focus:
+ha-consistency-invariants (umbrella contract) decomposed into ha-commit-engine,
+ha-commit-barrier-rollout, ha-single-leader-role, and ha-convergence-recovery.
+Supporting/foundational specs: ha-config-bootstrap, ha-heartbeat-liveness,
 ha-peer-connection-reuse, ha-repl-hardening, ha-wal-replication, raft-library,
 election-iomgr-refactor, scheduler-controller-msg-gap, worker-join-refactor,
-resval-id-decoupling.
+resval-id-decoupling. See `roadmap.md` for dependency order and status.
 
 Key design decisions already locked:
 - No `fork()` for snapshots (mutex inheritance deadlock in multithreaded controller)

.kiro/steering/roadmap.md

@@ -36,6 +36,8 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o
 - [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_
 - [ ] OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. _(FUTURE_WORK.md §5.)_
 - [x] steering sync — `ha-async-io.md` synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.
+- [ ] xce_test shell command support — add `!<command>` prefix to `xce_test` test scripts to execute arbitrary shell commands inline during test runs. Enables test scripts to drive environment setup/teardown (e.g., kill a node, write a file, run a helper) without calling out to a separate harness layer. Scope: parser change in the xce_test runner to detect `!`-prefixed lines, fork/exec the remainder as a shell command, fail the test on non-zero exit, capture stdout/stderr to the test log.
+- [ ] scheduler framework enhancement — strengthen the scheduler subsystem: (a) improve job scheduling policy extensibility (pluggable queue disciplines beyond FIFO), (b) add priority-aware dispatch, (c) improve resource-aware scheduling (CPU/memory/GPU slot tracking per host), (d) wire scheduler decisions through the commit barrier so scheduling state is replicated and survives failover. Dependencies: ha-consistency-invariants (commit barrier). Spec advised before implementation; promote via `/kiro-spec-init "scheduler-framework"` when ready.
 
 ## Superseded / Absorbed Items
 

.kiro/steering/structure.md

@@ -7,6 +7,8 @@
 ```
 src/
 ├── lib/           C — shared library (libxce): data structures, IPC, utilities
+│   ├── raft/      C — reusable Raft consensus module (raft.c/raft.h)
+│   └── testsuite/ C — libxce unit tests (test_*.c)
 ├── controller/    C — cluster controller daemon
 ├── worker/        C — per-node job execution daemon (xcecd)
 ├── scheduler/     C — job scheduling engine (plugin architecture)

.kiro/steering/tech.md

@@ -97,4 +97,4 @@ Early return on every error. Log with zap fields before returning.
 3. **CGO bridge** — `backend/` package wraps C library calls; unit tests run `CGO_ENABLED=0`, backend tests need `CGO_ENABLED=1` inside nix dev shell.
 4. **Swagger-first API docs** — `swaggo/swag` annotations on handlers; docs served at runtime by Gin.
 5. **Structured logging** — every log entry carries context via `zap.String()`, `zap.Error()`. Never use `fmt.Println` for operational output.
-6. **Single-writer HA** — controller is always one active primary; standby election via Raft-style consensus (term-based voting, freshness gates) in `raft.c`, with the single-writer constraint avoiding Raft log replication complexity.
+6. **Single-writer HA** — controller is always one active primary; standby election via Raft-style consensus (term-based voting, freshness gates) in the `src/lib/raft/` shared module (`raft.c`/`raft.h`), with the single-writer constraint avoiding Raft log replication complexity.