myxce/.kiro/specs/ha-prevote/research.md

# Gap Analysis: ha-prevote

_Generated by `/kiro-validate-gap`. Informational — presents options and research items, not final design decisions._

## Analysis Summary

- **The fix must be sender-side.** The disruptive step-down occurs in `raft_on_vote_request` Step 2 (raft.c:125-134) — term adoption + leader demotion — which runs **before** the `on_vote_eligible` gate (Step 5). No receiver gate today can save a healthy leader from a *fresh, higher-term* `RequestVote`, because the term is already adopted before any stickiness check. PreVote prevents the term inflation at the source.
- **Most receiver-side gating already exists.** `raft_cb_vote_eligible` (ha.c:217-245) already implements PRIMARY-deny, fresh-heartbeat-on-standby deny (leader stickiness), and catch-up deny; the dispatcher already enforces the freshness floor via `ha_vote_self_freshness`. PreVote evaluation can largely **reuse these gates** but must run with **zero state mutation**.
- **The new machinery is the candidate-side PreVote round** plus a wire message and a way to tally PreVote replies without touching `term`/`voted_for`.
- **Recommended shape: Hybrid (Option C).** New PreVote state in the raft library + new wire message/opcode, reusing the existing `ha_elect_ctx_t` collection, dispatcher freshness gate, and `raft_cb_vote_eligible` semantics.
- **Effort: M (3–7 days). Risk: Medium** — touches the safety-critical election path and the TLA+-referenced invariants, but integrates with well-factored, test-covered seams.

---

## 1. Current State Investigation

### Election path (raft library — `src/lib/raft/raft.c`, `raft.h`)

- **State** (`raft_state_t`, raft.h:23-38): `term`, `role` (`RAFT_FOLLOWER|CANDIDATE|LEADER`), `voted_for`, `voted_for_term`, `votes_granted`, `replies_received`, `election_decided`, `election_timeout_ms`, `my_priority`. **No pre-vote fields.**
- **Callbacks** (`raft_callbacks_t`, raft.h:66-96): `on_send_vote_request(ctx, peer_idx, term)`, `on_vote_eligible(ctx, from_idx, term)`, `on_become_leader`, `on_become_follower`, `on_persist(ctx, term, voted_for)`, `on_schedule_timeout`, `on_cancel_timeout`, `on_start_election_after_delay`, `on_trace`.
- **`raft_start_election`** (raft.c:61-108): increments `term`, sets self-vote, **persists**, then sends `on_send_vote_request` to each peer, schedules timeout. Single-node short-circuit promotes immediately.
- **`raft_on_vote_request`** (raft.c:110-191): 6-step flow — (1) stale-term deny; (2) **term upgrade → adopt term, clear vote, step down, fire `on_become_follower` if was LEADER** (the disruptive path); (3) LEADER role deny; (4) one-vote-per-term; (5) `on_vote_eligible` policy gate; (6) persist-then-grant.
- **`raft_on_vote_reply`** (raft.c:193-252): term-upgrade demotion; quorum = `(n_peers/2)+1`; all-replied-no-quorum fallback.
- **Triggers**: `raft_on_leader_lost` → `on_start_election_after_delay` (priority jitter); `raft_check_and_start_election` (raft.c:282-294) skips if already voted for another, else `raft_start_election`.

### Controller adapter (`src/controller/cluster.c`, `ha.c`, `ha.h`)

- **Dispatch** (`cluster.c` `OPCode_requestVote`, lines 359-408): parses `request_vote_req_t`, computes freshness floor via `ha_vote_self_freshness` (max of local and learned-committed tag/LSN), applies `freshness_ok`, maps candidate name → index, then calls `raft_on_vote_request`.
- **Candidate send** (`ha.c` `on_vote_connected`, lines 2579-2610): builds `request_vote_req_t` (term, candidate_id, config_tag, last_log_lsn), sends via `xce_request` (or `xce_vote_request_hook` under `XCE_TESTING`).
- **Collection** (`ha_elect_ctx_t`, ha.h:75-116): flexible-array `votes[]`, `votes_granted` (init 1 = self), `n_replied`, `max_resp_tag/lsn` (yield gate), `done` semaphore. `handle_request_vote_reply` (ha.c:2473-2528) tallies and calls `ha_elect_evaluate_and_signal`. Quorum = `(n_peers+1+1)/2`.
- **Voter gate** (`raft_cb_vote_eligible`, ha.c:217-245): PRIMARY → deny; STANDBY + `ha_heartbeat_is_fresh` → deny; in catch-up → deny; else eligible (freshness already enforced by dispatcher).
- **Leader stickiness primitive** (`ha_heartbeat_is_fresh`, ha.c:2013-2022): `(now - last_activity) < heartbeat_timeout_ms`. `ha_on_heartbeat_received` updates `last_activity`.
- **Election initiation** (`ha_start_election_async`, ha.c:3341-3410): allocates `ha_elect_ctx_t`, sets self-vote, stores `ha->elect_ctx`, delegates to `raft_start_election`.
- **Election triggers**: `ha_on_primary_ctrl_fd_rdhup` (ha.c:3486) on connection loss; `liveness_poll_fn` (ha.c:3684-3722) on heartbeat timeout.

### Wire schema (`src/schema/controller.capnp`)

```capnp
struct RequestVoteRequest {                 # lines 796-801
  term @0; candidateId @1; configTag @2; lastLogLsn @3;
}
struct RequestVoteReply {                    # lines 803-808
  term @0; voteGranted @1; peerConfigTag @2; peerLastLogLsn @3;
}
```
`OPCode` is an enum (lines 10-49); `requestVote @32` (line 43). **No PreVote opcode.**

### Test seams

- `xce_vote_request_hook` (ha.c:2560) intercepts the candidate send without live iomgr/SSL.
- `src/lib/testsuite/test_raft.c` — raft library unit tests with mock callbacks (TLA+ invariants).
- `src/controller/testsuite/`: `test_ha_election.c`, `test_ha_vote_gate.c`, `test_ha_start_election_async.c`, `test_ha_vote_send.c`, `test_ha_liveness.c`, `test_ha_role_lifecycle.c`, `test_ha_failover.c`.

### Relevant constants (`ha.h:121-129`)

`HEARTBEAT_INTERVAL_MS=3000`, `HEARTBEAT_TIMEOUT_MS=10000`, `ELECTION_TIMEOUT_MS=5000`, `ELECTION_JITTER_MAX_MS=200`, `ELECTION_PRIORITY_STEP_MS=500`.

---

## 2. Requirement-to-Asset Map

| Req | Capability needed | Existing asset | Gap |
|-----|-------------------|----------------|-----|
| **1** PreVote precedes term increment | A pre-election round that runs before `raft_start_election`; no persist/term change | `raft_start_election` (raft.c:61), `ha_start_election_async` (ha.c:3341) as the call site to gate | **Missing**: PreVote state + entry point; must interpose between trigger and `raft_start_election` |
| **1.2/1.3** No persistent mutation during PreVote | — | `on_persist` is only called by `raft_start_election`/`raft_on_vote_request` | **Constraint**: PreVote path must never call `on_persist` or mutate `term`/`voted_for` |
| **2** Voter eligibility reuses real-vote gate | Freshness + log-completeness check | Dispatcher freshness (`ha_vote_self_freshness`, cluster.c:389) + `raft_cb_vote_eligible` (ha.c:217) | **Constraint/Reuse**: run the same gate; must NOT record a vote |
| **2.3/2.4** No vote recorded; can still grant real vote later | — | one-vote-per-term lives in `raft_on_vote_request` Step 4 | **Constraint**: PreVote evaluation must bypass Steps 2/4/6 side effects |
| **3** Leader stickiness | Deny PreVote while leader believed alive; PRIMARY denies | `ha_heartbeat_is_fresh` (ha.c:2013), PRIMARY-deny in `raft_cb_vote_eligible` | **Reuse**: already present; just invoke from the PreVote receiver path |
| **3.1** Stickiness window semantics | "would not itself start an election" | Today: `heartbeat_timeout_ms` (10s) for freshness vs. `election_timeout_ms` (5s) probe | **Unknown (Research)**: which window defines stickiness — see RN-2 |
| **4** Frozen-then-resumed is non-disruptive | Candidate gets no PreVote majority → no term bump → re-attach | `ha_on_heartbeat_received` re-attach path | **Missing**: end-to-end behavior emerges from Req 1+3; needs integration test |
| **5** Legitimate changes still proceed | True death / partition heal still elect | `raft_on_vote_reply` term-upgrade demotion; existing failover (bug-339), rejoin (bug-340) | **Constraint**: do not regress; real-vote path unchanged after PreVote success |
| **6** Safety invariants | Election Safety, term monotonicity, Leader Completeness | `raft/raft.tla`, `test_raft.c`, controller verification checklist | **Constraint**: PreVote must be modeled/argued against TLA+; add invariant tests |
| **7.1** Transport compatibility | PreVote over existing election transport | capnp `OPCode`, `xce_request`, `peer_ctrl_msg_handler` | **Missing**: new opcode/flag + reply + dispatch case |
| **7.2/7.3** Observability | Log PreVote denials / abandoned campaigns | `on_trace` callback, `tlog` | **Missing**: trace/log points on PreVote deny + campaign abandonment |

Gap tags: **Missing** = no asset; **Constraint** = existing behavior bounds the design; **Reuse** = asset directly applicable; **Unknown** = research item.

---

## 3. Implementation Approach Options

### Option A — Extend the existing RequestVote path with an `isPreVote` flag
Add `isPreVote @4 :Bool = false` to `RequestVoteRequest` (and a matching flag/struct for the reply). Reuse the same dispatch case, same `on_vote_connected` send, and branch inside `raft_on_vote_request` / `raft_cb_vote_eligible` on the flag to skip all state mutation.

- ✅ Smallest wire change; backward-compatible capnp default; one RPC path.
- ✅ Reuses send/collect/gate machinery almost verbatim.
- ❌ Risks branching the safety-critical `raft_on_vote_request` with "if pre_vote skip steps 2/4/6" — easy to get a side-effect leak wrong.
- ❌ Overloads one function with two semantics (real vs. hypothetical).

### Option B — New PreVote opcode + dedicated PreVote functions
Add `preVoteRequest`/`preVoteReply` opcodes and `PreVoteRequest`/`PreVoteReply` structs; add `raft_start_pre_vote`, `raft_on_pre_vote_request`, `raft_on_pre_vote_reply`, and `raft_cb_pre_vote_eligible`; a parallel (or shared) `ha_elect_ctx_t` pre-vote tally.

- ✅ Clean separation; the pure (no-side-effect) PreVote functions are trivially auditable against the safety invariants.
- ✅ Real-vote path stays byte-for-byte unchanged (zero regression risk to bug-339/340).
- ❌ More new files/symbols and a second dispatch/collection path.
- ❌ Some duplication of freshness/eligibility logic unless carefully shared.

### Option C — Hybrid (recommended)
New PreVote **wire message + raft state-machine functions** (Option B's clarity for the safety-critical core), but **reuse** the controller-side assets: the dispatcher freshness computation, `raft_cb_vote_eligible` semantics (called read-only), the `ha_elect_ctx_t` collection (add a `pre_votes_granted` counter or a `phase` field rather than a whole second struct), and `xce_vote_request_hook` for tests.

- ✅ Keeps the pure PreVote decision isolated and auditable while avoiding duplicate collection/gate code.
- ✅ Real-vote path untouched; PreVote success simply calls the existing `raft_start_election`.
- ✅ Maps onto existing test seams (extend `test_ha_vote_gate.c`, add PreVote cases to `test_raft.c`).
- ❌ Requires a deliberate decision on shared-vs-separate eligibility helper to avoid drift.

---

## 4. Effort & Risk

| Dimension | Rating | Justification |
|-----------|--------|---------------|
| **Effort** | **M (3–7 days)** | New wire message + raft PreVote functions + candidate round + tests; but heavy reuse of existing gates, collection, and transport. |
| **Risk** | **Medium** | Touches the election safety core and TLA+-referenced invariants; mitigated by well-factored callback seams, existing gate logic to reuse, and strong test coverage. Highest-risk subtlety: ensuring the PreVote path has **zero** persistent side effects. |

---

## 5. Recommendations for Design Phase

**Preferred approach:** Option C (Hybrid).

**Key decisions to finalize in design:**
1. **Wire representation** — separate `preVoteRequest`/`preVoteReply` opcodes (recommended for auditability) vs. `isPreVote` flag on the existing struct. Either is capnp-compatible; the opcode keeps the real-vote path provably untouched.
2. **Eligibility sharing** — factor the freshness + `raft_cb_vote_eligible` checks into a read-only helper callable from both the real-vote and PreVote receiver paths, so the two cannot drift.
3. **Collection reuse** — extend `ha_elect_ctx_t` with a PreVote phase/counter rather than a second struct.
4. **PreVote success → real election** — on PreVote majority, call the **unchanged** `raft_start_election`; confirm the candidate re-checks leader-stickiness was still absent at the moment of starting the real round.
5. **Defense-in-depth question** — decide whether to *also* add leader-stickiness gating to the real `raft_on_vote_request` term-upgrade path (Raft "reject RequestVote within min election timeout"), since PreVote alone leaves Step 2 able to step down a leader if a stray higher-term real RequestVote ever arrives.

**Research Needed (carry to design):**
- **RN-1**: Model/argue the PreVote phase against `raft/raft.tla` and the controller verification checklist; confirm Election Safety + term monotonicity hold with the new path. (Req 6)
- **RN-2**: ✅ **RESOLVED — reuse `heartbeat_timeout_ms` via `ha_heartbeat_is_fresh()`.** See "Resolved Decisions → RN-2" below for the full comparison. (Req 3.1)
- **RN-3**: Confirm whether the PRIMARY should deny PreVotes via the existing `raft_cb_vote_eligible` PRIMARY-deny (Req 3.3) and that this composes with CheckQuorum-style behavior already present.
- **RN-4**: Decide PreVote round timeout/abandonment behavior and its interaction with the existing election jitter/priority and `election_timeout_ms` bounds (Req 5.4) — ensure no added unbounded delay.

**Explicitly out of scope:** Backward/rolling-upgrade compatibility with peers that predate PreVote. The product is **pre-release**, so all controllers in a cluster deploy from the same build — there is no mixed-version (old-binary ↔ new-binary) cluster to support. The wire change (new opcode or struct field) may be made without a negotiation/fallback path. Req 7.1's "without disrupting real RequestVote/RequestVoteReply exchanges" refers to PreVote coexisting with the *concurrent same-version* real-vote path, not to inter-version compatibility.

---

## 6. Resolved Decisions

### RN-2 — Leader-stickiness window → reuse `heartbeat_timeout_ms` (Opt 1)

**Decision:** The voter's leader-stickiness predicate is the existing `ha_heartbeat_is_fresh()` (`now − last_activity < heartbeat_timeout_ms`, currently 10s), gated on `role == CTRL_STANDBY`. PreVote introduces **no new timing concept**.

**The governing invariant.** Stickiness is a single voter-side predicate: *"do I still believe a leader is alive?"* It must be **pinned to the failure-detection window** — the period during which the voter would not itself start an election. If the two diverge:
- **stickiness < failure-detection** → a gap opens where the voter grants a PreVote while still believing its leader is alive and not yet failing over → **reopens the disruption hole**.
- **stickiness > failure-detection** → a voter that has already decided the leader is dead (and is campaigning) still denies others' PreVotes → mutual denial → **PreVote majorities can't form → liveness stall**.

So the window must equal whichever constant is XCE's *failure detector*.

**Naming trap.** Textbook Raft overloads one "election timeout" for two roles; XCE separates them, so the dissertation's "reject votes within the minimum election timeout" maps to `HEARTBEAT_TIMEOUT_MS`, **not** `ELECTION_TIMEOUT_MS`:

| Raft concept | XCE constant | Value | Role in XCE |
|---|---|---|---|
| election timeout *as leader-failure detector* | `HEARTBEAT_TIMEOUT_MS` | 10000 | Follower decides primary is dead → triggers failover (`liveness_poll_fn` / RDHUP) |
| election timeout *as vote-collection deadline* | `ELECTION_TIMEOUT_MS` | 5000 | Candidate waits this long for vote replies before giving up |
| heartbeat interval | `HEARTBEAT_INTERVAL_MS` | 3000 | Primary re-sends heartbeat every 3s |

**Option comparison.**

| | Opt 1: `heartbeat_timeout_ms` (10s) | Opt 2: `election_timeout_ms` (5s) | Opt 3: new `prevote_stickiness_ms` | Opt 4: `heartbeat_interval_ms` (3s) |
|---|---|---|---|---|
| Matches XCE failure-detector | ✅ exactly | ❌ wrong constant (vote-collection deadline) | ⚠️ only if set == 10s | ❌ far shorter |
| Consistent w/ existing real-vote gate (`raft_cb_vote_eligible`) | ✅ same predicate reused | ❌ real-vote 10s vs PreVote 5s drift | ⚠️ drift unless tuned | ❌ drift |
| Disruption gap (grant while leader alive) | ✅ none | ❌ [5s,10s] gap on flaky/asymmetric link | ⚠️ gap if < 10s | ❌ [3s,10s] gap |
| Tolerance to transient heartbeat loss | ✅ ~3 missed heartbeats | ⚠️ ~1 missed heartbeat | ⚠️ depends | ❌ 0 missed |
| Liveness-stall risk | ✅ none | ✅ none | ⚠️ stall if > 10s | ✅ none |
| Speeds up legitimate failover | n/a (matches today) | ❌ no real benefit¹ | ⚠️ only by also lowering failover timeout | ❌ no |
| New config surface | none | none | +1 tunable | none |

¹ Under symmetric silent death nobody campaigns until the 10s failure-detection fires, so a shorter stickiness window has no candidate to grant to in [5s,10s] — zero speedup. A real crash closes sockets → RDHUP → standbys leave `CTRL_STANDBY` and campaign immediately, independent of the window. The shorter window's only observable effect is the disruption gap on a flaky link (a healthy primary missing ~2 of 3 heartbeats to one voter would get a PreVote granted against it at ~6s).

**Why Opt 1.** It is the single value where the stickiness window, the failure-detection trigger (`liveness_poll_fn`), and the existing real-vote stickiness gate (`raft_cb_vote_eligible`) all key off **one predicate** (`ha_heartbeat_is_fresh`). Gating on `role == CTRL_STANDBY` also handles connection loss for free: RDHUP moves the node out of `CTRL_STANDBY`, so a genuinely disconnected voter stops being sticky immediately instead of waiting the full 10s. This is exactly what Req 3.1's observable phrasing ("received leader traffic recently enough that it would not itself start an election") already describes. If failover latency is later reduced by lowering `HEARTBEAT_TIMEOUT_MS`, stickiness follows automatically — which is itself the argument against Opt 3's separable constant.

**Requirement impact:** Req 3.1 can be tightened in design (Boundary Commitment) to name `ha_heartbeat_is_fresh` / `heartbeat_timeout_ms` as the binding, while keeping the requirement text observable.

---

## 7. Design Synthesis Outcomes (design phase)

Applied the generalization / build-vs-adopt / simplification lenses before writing `design.md`.

### Generalization — share interfaces, not implementations
PreVote and real-vote are two variations of "solicit a vote decision." Two seams are shared rather than duplicated:
- **Voter eligibility** — `raft_cb_vote_eligible` already encodes exactly the PreVote gates (PRIMARY-deny + fresh-heartbeat stickiness + catch-up-deny). The PreVote receiver reuses it verbatim via the existing `on_vote_eligible` callback; the only PreVote-specific rule is "decide without adopting the term."
- **Candidate tally** — `ha_elect_ctx_t` is reused with a `phase` discriminator (`HA_ELECT_PREVOTE` / `HA_ELECT_REAL`) instead of a second collection struct. Same quorum formula `(n_peers + 2) / 2`, same `votes[]` idempotency.

The raft state-machine *functions* stay distinct (`raft_on_pre_vote_*` vs `raft_on_vote_*`) so the pure, no-side-effect PreVote path is independently auditable against the TLA+ model.

### Build vs. Adopt — adopt
- **Algorithm**: Raft dissertation §9.6 PreVote (standard, well-specified). Not invented.
- **Infrastructure**: capnp election transport, iomgr/delaymgr, `ha_elect_ctx_t`, `ha_heartbeat_is_fresh`, `ha_vote_self_freshness`, `xce_vote_request_hook`. No new libraries.

### Simplification decisions
- **Wire**: new `preVoteRequest`/`preVoteReply` **opcodes that reuse the existing `RequestVoteRequest`/`RequestVoteReply` structs and union arms** — no new capnp structs, and the real-vote handler stays byte-for-byte unchanged (zero regression risk to bug-339/340). Chosen over an `isPreVote` flag on the existing struct (which would branch the safety-critical real-vote handler).
- **Defer defense-in-depth** (the RN-4-adjacent / open "real-vote stickiness guard"): PreVote guarantees a candidate only sends a real RequestVote *after* a majority's stickiness has lapsed, so in a same-version cluster no stickiness-violating real RequestVote can arise. Adding the guard would touch the real-vote term-adoption path for no in-scope benefit. **Deferred**, documented in `design.md` → Architecture → Deferred decisions. (Resolves the open defense-in-depth question; RN-4 reduces to "bound the PreVote round with `election_timeout_ms` and re-arm liveness on loss," which the design specifies.)
- **No new role state**: the candidate stays `CTRL_STANDBY` during the PreVote round (tracked by `ctx.phase`), so a lost PreVote needs no rollback. The destructive `truncate-WAL + CTRL_ELECTING` step is **moved** from the failover trigger to PreVote success — the key change that makes a transient freeze a true no-op (Req 4).

### Remaining open research item
- **RN-1**: TLA+ / verification-checklist argument that Election Safety + term monotonicity hold with the PreVote path (the real-vote path is unchanged, so this is largely an argument that the pure PreVote functions mutate nothing). Carried into implementation as `test_raft.c` invariant assertions. RN-3 (PRIMARY-deny composition) is resolved by reusing `raft_cb_vote_eligible`.

---

## RN-1 resolved (task 6.1)

**Decision/outcome:** The PreVote phase is consistent with the repository's
`raft/raft.tla` reference and the controller verification checklist —
Election Safety (≤1 leader per term), Leader Completeness, term monotonicity,
and a monotonic commit index all continue to hold with PreVote enabled.

**Argument (purity ⇒ no new transitions).**

1. **The PreVote functions are pure w.r.t. persistent Raft state.** Across an
   arbitrary sequence of `raft_begin_pre_vote` / `raft_on_pre_vote_request` /
   `raft_on_pre_vote_reply` calls — mixed grant+deny, duplicate replies,
   out-of-range peers, and both WON and LOST terminal outcomes — none of them
   ever writes `s->term`, `s->voted_for`, `s->voted_for_term`, or `s->role`,
   and none invokes `on_persist`. Mechanical evidence:
   `test_invariant_prevote_sequence_pure_no_persist` and
   `test_invariant_prevote_lost_sequence_pure_no_persist` in
   `src/lib/testsuite/test_raft.c`.

2. **Term monotonicity is preserved.** Because the PreVote path writes no term,
   the only writers of `s->term` remain the UNCHANGED `raft_start_election`
   (the real, gated term bump) and `raft_on_vote_request` Step 2 (adopt a
   strictly higher term). PreVote can only *gate* whether a real election
   later runs; it never advances a term itself, and it never decreases one.

3. **Election Safety is preserved.** Because the PreVote path creates no leader
   (no `on_become_leader`, no `role := RAFT_LEADER`), it adds no new way to
   satisfy the antecedent of Election Safety. A PreVote round reaching
   `RAFT_PRE_VOTE_WON` returns a *status* only; the node stays `RAFT_FOLLOWER`
   until the separate, unchanged `raft_start_election` / `raft_on_vote_reply`
   path runs. Evidence: `test_invariant_prevote_won_grants_no_leadership`.

4. **The bug-339/bug-340 failover path is unchanged.** The legitimate
   higher-term-adopt + leader-step-down behavior lives entirely in
   `raft_on_vote_request` / `raft_on_vote_reply`, which this feature does not
   touch. PreVote sits strictly before that path and (by 1) leaves its inputs
   untouched. Evidence: `test_invariant_real_vote_higher_term_adopt_unchanged`
   demonstrates the real-vote path still adopts a higher term, fires
   `on_become_follower` to step a leader down, and elects at quorum via
   `on_become_leader` — even after a full PreVote round has run on the node.

**Conclusion.** PreVote introduces no transition that writes a term or creates
a leader; every term-writing / leadership-creating transition in the model is
byte-for-byte the pre-PreVote transition. The set of reachable
`(term, leader)` configurations is therefore unchanged, so the `raft/raft.tla`
invariants (Election Safety, Leader Completeness, term monotonicity, monotonic
commit index) and the controller verification checklist continue to hold. The
four `test_invariant_*` tests are the mechanical evidence for premises (1)–(4),
and the existing failover/rejoin regression suites remain green with PreVote
enabled (controller suite 563 pass / 0 fail after task 4.2). Requirements
covered: 5.2, 5.3, 6.1, 6.2, 6.3, 6.4.

---

## Gap Re-Audit (2026-06-18)

**Method.** Verified every EARS acceptance criterion in `requirements.md` against
the current on-disk source via codegraph + grep. tasks.md shows phase
`tasks-generated` with 6 unchecked `[ ]` and 12 checked `[x]`; this audit
classifies each `[ ]` and tags each requirement MET/Missing/Unknown/Constraint.

### STATUS: CLOSED — requirements met 7/7, all 12 leaf tasks implemented + tested

Every production seam the design names exists, is wired into the real failover
path, and is exercised by tests. The 6 unchecked boxes are **group-header
artifacts**, not work.

### Open-task classification (the 6 unchecked `[ ]`)

All 6 unchecked boxes are the **phase/group-header lines** (task numbers `1`–`6`,
the section titles). Every one of their leaf sub-tasks is checked `[x]`:

| `[ ]` header | Children (all `[x]`) | Classification |
|---|---|---|
| 1. Foundation: wire schema + raft scaffolding | 1.1, 1.2 | group-header-artifact |
| 2. Core: raft-library pure decision + tally | 2.1, 2.2, 2.3 | group-header-artifact |
| 3. Core: controller voter-side dispatch | 3.1 | group-header-artifact |
| 4. Core: candidate orchestration + trigger restructure | 4.1, 4.2 | group-header-artifact |
| 5. Integration: transport wiring + E2E | 5.1, 5.2, 5.3 | group-header-artifact |
| 6. Validation: safety invariants + regression guard | 6.1 | group-header-artifact |

**Genuine-gap: 0. Deferred: 0. Artifact: 6.** A group header is complete by
definition once all its children are done; these were never intended as
separately-checkable work items. No effort/risk applies (nothing to build).

### Requirement-to-Asset map

| Req | Criterion (summary) | Asset (file:line) | Tag |
|---|---|---|---|
| 1.1 | PreVote before term increment on trigger | `ha_begin_failover` ha.c:4606; `ha_start_prevote_async` ha.c:4193; triggers `ha_on_primary_ctrl_fd_rdhup` ha.c:4639 / `liveness_poll_fn` (calls rdhup) ha.c:~4789 | MET |
| 1.2 | No term/vote mutation during round | `raft_begin_pre_vote` raft.c:232 (writes only `pre_vote_*` tally; no term/voted_for/role/persist) | MET |
| 1.3 | Request prospective term+1, not persisted | `raft_begin_pre_vote` raft.c:243 (`prospective_term = s->term + 1`, local only) | MET |
| 1.4 | No majority → no real election | `pre_vote_reply_dispatch` ha.c:4296 LOST branch (stay STANDBY, re-arm) | MET |
| 1.5 | Majority → start real election | `pre_vote_reply_dispatch` WON → `ha_enter_real_election` ha.c:4103 → unchanged `ha_start_election_async`/`raft_start_election` | MET |
| 2.1 | Grant only if real-vote freshness gate passes | cluster.c:480 `ha_vote_self_freshness` floor + cluster.c:512 `raft_on_pre_vote_request` | MET |
| 2.2 | Fail freshness → deny | cluster.c:482-506 `freshness_ok` → `deny_reason="freshness"` | MET |
| 2.3 | Grant/deny records no vote, no term | `raft_on_pre_vote_request` raft.c:193 (pure); reply carries unchanged term cluster.c:500/556 | MET |
| 2.4 | PreVote does not consume the real vote | `raft_on_pre_vote_request` writes no `voted_for`; asserted `test_genuine_failover...` test_ha_prevote.c:709 | MET |
| 3.1 | Deny while leader believed alive | `raft_cb_vote_eligible` ha.c:299 (STANDBY + fresh heartbeat + primary_ctrl_fd>=0 → deny), reused by PreVote | MET |
| 3.2 | Past self-election threshold → freshness-only eval | `raft_cb_vote_eligible` falls through to grant once heartbeat stale; `test_genuine_failover_stale_peer_grants_and_elects` | MET |
| 3.3 | Primary denies PreVotes | `raft_cb_vote_eligible` ha.c:307 (CTRL_PRIMARY → 0) | MET |
| 4.1 | Frozen-resumed gets no majority | scenario test `test_ha_prevote.c` (Angle B deny) + cluster.c deny path | MET |
| 4.2 | Re-attach on next heartbeat, no term change | `ha_on_heartbeat_received` ha.c:4886-4909 (cancel round mid-flight, stay STANDBY); `test_heartbeat_mid_round_cancels_prevote` test_ha_failover_prevote.c:198 | MET |
| 4.3 | Healthy primary not demoted/restarted | trigger defers truncate/CTRL_ELECTING to WON only (ha.c:4655-4662) | MET |
| 4.4 | Term unchanged, no re-election cycle | candidate stays CTRL_STANDBY through round; round state in `raft_state_t` only | MET |
| 5.1 | True death → PreVotes granted → elect | `test_genuine_failover_stale_peer_grants_and_elects` test_ha_prevote.c:680 | MET |
| 5.2 | Partition heal → adopt higher term | `raft_on_vote_reply` raft.c:346 (unchanged real-vote path) | Constraint (unchanged, preserved) |
| 5.3 | No regression of bug-339/bug-340 | real-vote path byte-for-byte unchanged; `test_invariant_real_vote_higher_term_adopt_unchanged` (RN-1) | MET |
| 5.4 | Complete within election-timeout bounds | round timeout `election_timeout_ms` ha.c:4258; rdhup disarms liveness first | MET |
| 6.1 | Election Safety preserved | PreVote fires no `on_become_leader`; RN-1 + `test_invariant_prevote_won_grants_no_leadership` | MET |
| 6.2 | Term monotonicity; PreVote never changes term | RN-1 + `test_invariant_prevote_sequence_pure_no_persist` | MET |
| 6.3 | Leader Completeness, monotonic commit index | freshness floor reused; commit path untouched | Constraint (unchanged, preserved) |
| 6.4 | Consistent with raft.tla + checklist | RN-1 argument (research.md §RN-1) + invariant tests | MET |
| 7.1 | PreVote over existing transport, no real-vote disruption | new opcodes `preVoteRequest @38`/`preVoteReply @39` controller.capnp:49-50, reuse vote structs; reply routed via `peer_ctrl_msg_handler` ha.c:3610 (pre_vote_in_progress gate) → `pre_vote_reply_decode_and_dispatch` ha.c:3340 | MET |
| 7.2 | Log denials w/ candidate + reason | cluster.c:546-552 `INFO("HA: PreVote DENIED candidate=... reason=...")` (gate-order-faithful reason) | MET |
| 7.3 | Log abandoned campaign distinctly | `pre_vote_reply_dispatch` LOST branch ha.c:4313 INFO; round-timeout fn | MET |

**Tally: MET 25, Constraint (preserved-unchanged) 2 (5.2, 6.3), Missing 0, Unknown 0.**

### Notable implementation facts (beyond the design baseline)

- **Reply routing** is NOT a dedicated `OPCode_preVoteReply` switch case in
  cluster.c (as task 5.1 literally phrased it). Instead `peer_ctrl_msg_handler`
  (ha.c:3600) routes by role+state: `CTRL_STANDBY && pre_vote_in_progress` →
  `pre_vote_reply_decode_and_dispatch` → `pre_vote_reply_dispatch`. The request
  side does emit/reply over the dedicated `OPCode_preVoteReply` opcode
  (cluster.c:564). Functionally equivalent to the design intent; the dispatch
  discriminator is round-state rather than a third reply opcode case.
- **Two extra robustness mechanisms** were added beyond the spec, both benign
  and in-scope-adjacent: (a) a **per-node stagger** (`ha_election_stagger_ms`
  → `ha_failover_stagger_fn`, ha.c:4586) before starting PreVote to avoid
  lock-step split votes (bug-606); (b) a retained **config-less-node guard**
  (tag==0 && n_peers>1 → voter-only, ha.c:4693) that short-circuits before the
  round (ha-config-bootstrap Req 2.8).
- **Test coverage** spans three files: `src/lib/testsuite/test_raft.c` (pure
  decision/tally + 4 invariant tests), `src/controller/testsuite/test_ha_prevote.c`
  (voter decide + frozen-rejoin + genuine-failover scenarios),
  `src/controller/testsuite/test_ha_failover_prevote.c` (trigger rewire:
  rdhup/liveness start PreVote not election, heartbeat-mid-round cancel,
  WON→real-election, config-less-node no-PreVote). Per Implementation Notes
  (task 4.2): controller suite 563 PASS / 0 FAIL with PreVote live.

### Carry-forward (NOT a ha-prevote gap)

The deliberate per-round `elect_ctx` ASAN leak (`ha_start_prevote_async`
allocates a metadata `ha_elect_ctx_t` that is intentionally not freed because
live vote endpoints may hold a bare pointer — same UAF caution as the rdhup
path) is pre-existing, bounded (~one per round), and explicitly carried into a
separate follow-up spec `.kiro/specs/ha-elect-ctx-lifecycle/`. It is out of
scope for ha-prevote and does not affect any ha-prevote requirement.

### Options for the (zero) genuine remaining work

- **Option A (recommended):** Mark ha-prevote CLOSED. Optionally check the 6
  group-header boxes for bookkeeping cleanliness (cosmetic; all children done).
  **Effort: S. Risk: none.**
- **Option B:** Run `/kiro-verify-completion ha-prevote` to mechanically confirm
  every acceptance criterion against the suite before flipping spec.json phase
  to a completed state. **Effort: S. Risk: low** (known `test_commit_barrier_reply`
  300s TIMEOUT in the full controller suite — use the targeted run and check for
  zero `:FAIL`, per Implementation Note 3.1).
- **Option C:** No action; treat the unchecked headers as a tasks.md rendering
  convention. **Effort: none. Risk: none** (the only downside is `[ ]` headers
  reading as "incomplete" to a casual `/kiro-spec-status` glance).