myxce/.kiro/specs/ha-prevote/tasks.md

Implementation Plan

TDD is mandatory (project rule): for each implementation sub-task, add the failing test in the relevant testsuite/ first (RED), then implement to green. Dependency direction: capnp/types → raft library → controller adapter → dispatch/transport. The real-vote state machine is reused unchanged after PreVote success.

• 1. Foundation: wire schema and raft-library PreVote scaffolding

• 1.1 Add PreVote opcodes to the election wire protocol

  • Add two opcodes for the PreVote request and PreVote reply, reusing the existing vote request/reply payload structures and their message union arms (no new payload structs).
  • Regenerate the generated bindings and confirm the build produces the new opcode constants while existing vote opcode ordinals are unchanged.
  • Requirements: 7.1
  • Boundary: PreVote dispatch (wire schema)

• 1.2 (P) Add PreVote types, state, and callback seam to the raft library

  • Add PreVote outcome codes (won / lost / pending), a self-contained PreVote tally on the raft state (granted count, per-peer replied tracking, in-progress flag), and an optional send-pre-vote callback in the callback table.
  • Header compiles with the new declarations and existing election state/layout is unchanged; no behavior added yet.
  • Requirements: 1.2, 6.2
  • Boundary: raft pre-vote functions

• 2. Core: raft-library pure PreVote decision and tally (test-first, no side effects)

• 2.1 Implement the voter-side PreVote decision

  • RED first: add raft unit tests asserting grant for a fresh, higher-or-equal-term candidate when the eligibility callback allows it, and deny on stale term or when the eligibility callback rejects.
  • Reuse the existing vote-eligibility callback (primary-deny, fresh-heartbeat stickiness, catch-up deny) and let the dispatcher apply freshness beforehand, exactly as the real-vote path does.
  • Observable: after a granted PreVote the test confirms term, voted-for, and role are all unchanged and no persist callback fired.
  • Requirements: 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 6.2
  • Boundary: raft pre-vote functions

• 2.2 Implement the candidate-side PreVote start

  • RED first: add a raft unit test asserting the prospective term used is current+1, that it is never written to the persistent term and never persisted, and that the send-pre-vote callback fires once per peer with a self-grant recorded.
  • Observable: test confirms the persistent term/voted-for are byte-for-byte unchanged after starting the round.
  • Requirements: 1.1, 1.2, 1.3, 6.2
  • Boundary: raft pre-vote functions
  • Depends: 1.2

• 2.3 Implement the PreVote reply tally

  • RED first: add raft unit tests asserting the tally lives on the raft state, returns won at quorum, lost when all peers reply without quorum, pending otherwise, and ignores duplicate replies from the same peer.
  • Mirror the production real-vote tally pattern, but never fire become-leader/become-follower — return a status the caller acts on.
  • Observable: test confirms role and term unchanged across won/lost/pending outcomes.
  • Requirements: 1.4, 1.5, 6.2
  • Boundary: raft pre-vote functions
  • Depends: 1.2

• 3. Core: controller voter-side PreVote dispatch (test-first)

• 3.1 Answer incoming PreVote requests with stickiness + freshness, no mutation

  • RED first: extend the vote-gate tests so a PreVote from a fresh candidate is denied while a leader is believed alive (primary role, or standby with fresh heartbeat, or catch-up) and granted once stickiness lapses and freshness passes.
  • Compute the freshness floor and run the eligibility gate before the raft decision, reusing the real-vote dispatch shape; reply with the voter's unchanged term plus the grant bit and freshness fields; log every denial with candidate id and reason.
  • Observable: a denied PreVote reply carries the voter's original term (no term adoption) and a diagnosable denial-reason log line is emitted.
  • Requirements: 2.1, 2.2, 3.1, 3.2, 3.3, 7.1, 7.2
  • Boundary: PreVote dispatch
  • Depends: 1.1, 2.1

• 4. Core: controller candidate-side orchestration and trigger restructuring (test-first)

• 4.1 (P) Run the PreVote round and branch on its outcome

  • RED first: add controller tests driving the reply handler through the existing vote-reply hook (no real transport): majority grant → real election started; no majority → node stays standby with term unchanged and the liveness watchdog re-armed; round timeout treated as a loss.
  • Owns the candidate orchestration in the controller: start the round (no persist, no role change), arm a round timeout bounded by the existing election-probe deadline, and the PreVote-reply handler that feeds the raft tally and branches — on a win perform the existing truncate + electing-transition + start-real-election sequence (unchanged real-election path); on loss log campaign abandonment. The send path emits the PreVote request opcode from task 1.1; transport-side reply routing is wired separately in task 5.1.
  • Observable: on no-majority the cluster term is unchanged, role stays standby, and a subsequent genuine failure still triggers a new round.
  • Requirements: 1.1, 1.4, 1.5, 5.4, 7.3
  • Boundary: PreVote orchestration
  • Depends: 1.1, 2.2, 2.3 (parallel-safe with 3.1: orchestration lives in a different file than the voter-side dispatch)

• 4.2 Route failover triggers through PreVote and defer the destructive step

  • RED first: add tests asserting the failover trigger no longer truncates the WAL or increments the term until PreVote wins, and that an incoming primary heartbeat mid-round cancels the round and re-attaches as standby.
  • Make both failover entry points (connection RDHUP and heartbeat-timeout poll) begin the PreVote round instead of starting an election directly; move truncate + electing-transition to PreVote success; cancel an in-progress round on heartbeat receipt.
  • Observable: a frozen-then-resumed standby that loses PreVote leaves the WAL and term untouched and re-attaches when the next heartbeat arrives.
  • Requirements: 1.1, 4.2, 4.3, 4.4
  • Boundary: PreVote orchestration (trigger path)
  • Depends: 4.1

• 5. Integration: transport wiring and end-to-end scenarios

• 5.1 Wire the PreVote reply opcode case to the candidate handler

  • Add the transport dispatch case for the incoming PreVote-reply opcode so it routes to the reply handler built in 4.1 (replacing the test hook in production), paralleling the real-vote reply dispatch case.
  • Observable: a PreVote reply arriving over the real transport reaches the candidate's tally and can drive the round to a decision end-to-end (not just via the test hook).
  • Requirements: 7.1
  • Boundary: PreVote dispatch
  • Depends: 1.1, 3.1, 4.1 (shares the dispatch file with 3.1; owns only the opcode-to-handler routing, while 4.1 owns the handler)

• 5.2 Frozen-then-resumed no-op scenario test

  • Add a 3-node scenario test: primary alive, a resumed standby starts a PreVote round; both peers deny (primary role + fresh heartbeat).
  • Observable: no real vote request is sent, the cluster term is unchanged across the event, and the primary stays primary (no demotion/truncate/self-restart).
  • Requirements: 4.1, 4.3, 4.4
  • Depends: 3.1, 4.1, 4.2, 5.1

• 5.3 Genuine-failover and re-attach scenario tests

  • Add scenario tests: dead primary with stale peer heartbeats → PreVotes granted → a new primary is elected within the existing election-timeout bounds; heartbeat mid-round → clean re-attach; failed PreVote → liveness watchdog re-armed so a later real failure still elects.
  • Observable: a genuine primary death elects a new primary, and the suppressed cases never do.
  • Requirements: 5.1, 5.4, 4.2
  • Depends: 4.1, 4.2, 5.1

• 6. Validation: safety invariants and regression guard

• 6.1 Assert Raft safety invariants hold with PreVote

  • Add raft-library invariant assertions that the PreVote functions mutate no persistent state (Election Safety and term monotonicity preserved), and record the TLA+/verification-checklist consistency argument (the real-vote path is unchanged, so the higher-term-adopt failover and the bug-339/bug-340 fixes are preserved).
  • Observable: invariant tests pass and the existing failover/rejoin regression tests remain green with PreVote enabled.
  • Requirements: 5.2, 5.3, 6.1, 6.2, 6.3, 6.4
  • Depends: 2.1, 2.2, 2.3

Implementation Notes

• (1.1) Wire opcodes preVoteRequest @38 / preVoteReply @39 reuse existing RequestVoteRequest/RequestVoteReply structs and union arms (requestvotemsg @27 / requestvotereplymsg @14). Generated bindings live under src/xce.build/lib/controller.capnp.h (build artifacts, not source-committed).
• (1.2) Raft PreVote scaffolding in raft.h: outcome codes RAFT_PRE_VOTE_WON/LOST/PENDING = 7/8/9; tally fields pre_vote_in_progress, pre_votes_granted, pre_vote_replies[RAFT_MAX_PEERS] appended to raft_state_t (after my_priority); optional on_send_pre_vote_request callback (NULL-checkable). New #define RAFT_MAX_PEERS 8 bounds the per-peer dedup array — safe: production cap is MAX_HA_CANDIDATES=4 (src/include/config.h:78), and it matches RAFT_SIM_MAX_NODES=8. Tasks 2.1/2.2/2.3 supply the function bodies and MUST keep them pure (no term/voted_for/role mutation, no on_persist). Build/test the raft layer with cd src && nix develop --command bash tools/xce-build.sh lib.
• (3.1) IMPORTANT for controller tasks: cluster.c is NOT linked into the unit-test binary (controller_unit_test_src in src/controller/testsuite/meson.build) — it's daemon-only (controller_src). So dispatch cases in cluster.c can't be unit-tested directly; test their composed primitives + contract asserts, and defer end-to-end to scenario tasks 5.2/5.3. ha.c IS unit-test-linked (the test_ha_* suites exercise it). Controller build/test: cd src && nix develop --command bash tools/xce-build.sh controller; the registered meson test name is "controller test" (single Unity binary). Known pre-existing test_commit_barrier_reply 300s TIMEOUT makes the full suite exit non-zero — validate via the targeted run nix develop --command meson test -C xce.build --no-rebuild --print-errorlogs "controller test" and check for zero :FAIL (Timeout:1 is the known one). The new prevote opcodes' generated constants are OPCode_preVoteRequest=38 / OPCode_preVoteReply=39.
• (3.1, LOW nit) cluster.c PreVote denial LOG reason uses elimination order (stale_term→primary→catch_up→fresh_heartbeat) which differs from raft_cb_vote_eligible's gate order (primary→fresh_heartbeat→catch_up); in the rare STANDBY catch-up+fresh-heartbeat overlap the log string may say catch_up while the gate denied on fresh_heartbeat. Decision is identical (deny); log-string only. RESOLVED (commit 47f7338): exposed pure ha_heartbeat_is_fresh and rewrote the elimination to mirror the gate's exact order (stale_term → primary → fresh_heartbeat → catch_up).
• (4.1) ha.c now has ha_start_prevote_async, pre_vote_reply_dispatch, ha_enter_real_election (WON branch = the factored truncate+CTRL_ELECTING+ha_start_election_async helper), a raft_cb_send_pre_vote_request callback (registered in s_real_cbs), and a round-timeout task. KEY for task 5.1: pre_vote_reply_dispatch(ha, peer_idx, granted) takes the ABSOLUTE cluster peer index (matches raft_begin_pre_vote's send loop), NOT the real-vote local votes[] index. pre_vote_reply_dispatch is __attribute__((used)) with NO production caller yet — task 5.1's cluster.c OPCode_preVoteReply case must call it (replacing the xce_* test seam in production). Triggers (rdhup/liveness/heartbeat) are still UNMODIFIED — task 4.2 rewires them to call ha_start_prevote_async.
• (4.2) Triggers now rewired: ha_on_primary_ctrl_fd_rdhup + liveness_poll_fn → ha_begin_failover → ha_start_prevote_async (PreVote first; truncate+CTRL_ELECTING deferred to ha_enter_real_election on WON). ha_on_heartbeat_received cancels an in-progress round (only when pre_vote_in_progress) + re-attaches CTRL_STANDBY, no term change. LOAD-BEARING: ha_start_prevote_async checks raft_begin_pre_vote's return and on immediate RAFT_PRE_VOTE_WON (single-node, n_peers==1) synchronously calls ha_enter_real_election — without this a single-node/immediate-majority node never elects. Full controller suite 563 PASS/0 FAIL with PreVote live. (NOTE: pre-existing deliberate elect_ctx ASAN leak ~one-per-round carried into PreVote; out of scope. Tracked as follow-up spec .kiro/specs/ha-elect-ctx-lifecycle/ — pre-existing + bounded + UAF-sensitive carrier-lifetime fix, deferred by decision.)