myxce/.kiro/specs/ha-prevote/spec.json

{
  "feature_name": "ha-prevote",
  "created_at": "2026-06-15T23:45:43Z",
  "updated_at": "2026-07-18T09:55:00Z",
  "language": "en",
  "phase": "implementation-complete",
  "approvals": {
    "requirements": {
      "generated": true,
      "approved": true
    },
    "design": {
      "generated": true,
      "approved": true
    },
    "tasks": {
      "generated": true,
      "approved": true
    }
  },
  "ready_for_implementation": true,
  "implementation": {
    "completed": true,
    "tasks_done": "1.1, 1.2, 2.1, 2.2, 2.3, 3.1, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1 (all 12 leaf tasks; the 6 unchecked tasks.md group-header lines are bookkeeping artifacts, not work -- see research.md's 2026-06-18 Gap Re-Audit)",
    "notes": "PreVote phase (Raft dissertation Sec 9.6) added ahead of the failover trigger: raft_begin_pre_vote/raft_on_pre_vote_request/raft_on_pre_vote_reply (raft.c, pure -- no term/voted_for/role mutation, no persist) plus ha_begin_failover/ha_start_prevote_async/pre_vote_reply_dispatch/ha_enter_real_election (ha.c) orchestrating the round and deferring truncate+CTRL_ELECTING to PreVote success. Original implementation landed 2026-06-16 (commits 0d66026..e6b234a, controller suite 563/0 clean); research.md's own RN-1 TLA+/verification-checklist argument and 2026-06-18 gap re-audit closed all 25 MET + 2 preserved-unchanged criteria with zero gaps. Substantial hardening landed on top since: bug-605 (dispatch-gate wiring), bug-606 (per-node failover stagger to avoid lock-step split vote), bug-660 (dead-primary fresh-heartbeat convergence), bug-1421/1424 (PreVote-reply registration on connect/monitor failure), bug-1430 (UAF CAS fix in ha_prevote_round_timeout_fn), bug-1435 (raft_lock serialization), task 5.7 jittered [T,2T] election-timeout backoff, and ha-elect-ctx-lifecycle's reference-counted elect_ctx carrier (commit 6529dcf, replacing the deliberate per-round leak this spec's tasks.md Implementation Notes had carried forward as a known, deferred limitation -- now resolved by that sibling spec, itself pending its own validate-impl pass)."
  },
  "validate_impl": {
    "decision": "GO",
    "validated_at": "2026-07-18T09:55:00Z",
    "note": "Cross-task/feature-level validation of a spec whose 18/18 tasks.md checkboxes (12 leaf + 6 group-header) had been complete since 2026-06-16, but whose spec.json phase was never advanced past tasks-generated (same administrative-oversight pattern as several sibling specs validated earlier today: ha-wal-replication, ha-heartbeat-committed-propagation). Ran immediately after raft-drive-async (NO-GO, bug-1447 ASAN SEGV in src/lib/netio.c SSL handshake, round 9/10 of its own sequential soak) and after ha-wal-replication / ha-heartbeat-committed-propagation (both GO, clean 10/10 soaks) -- extra weight was given to reproducing or ruling out bug-1447's signature in this spec's own live soak, especially since round 9 is the exact round bug-1447 hit.\n\nCROSS-TASK CONSISTENCY / DESIGN ALIGNMENT (codegraph-verified against design.md's Architecture, Boundary Commitments, and Requirements Traceability table, plus research.md's own 2026-06-18 gap re-audit which independently mapped all 27 acceptance criteria to file:line evidence): confirmed live on-disk source for raft_begin_pre_vote/raft_on_pre_vote_request/raft_on_pre_vote_reply (src/lib/raft/raft.c) and ha_begin_failover/ha_start_prevote_async/pre_vote_reply_dispatch/ha_enter_real_election (src/controller/ha.c) all still present with the designed pure/side-effect-free shape (zero term/voted_for/role mutation, zero on_persist calls in the three raft-library functions -- verified by direct source read, not just test titles). ha_dispatch_gate (ha.c:3794) admits OPCode_preVoteRequest/preVoteReply in CTRL_INIT/CTRL_ELECTING/CTRL_STANDBY exactly as bug-605's fix requires. Confirmed the dependency direction (capnp/types -> raft library (pure) -> controller adapter -> dispatch/transport) still holds with no upward imports. One legitimate, expected architectural evolution beyond the original design.md snapshot: raft_on_pre_vote_request's signature grew cand_last_log_term/cand_last_log_index parameters and now runs the (lastLogTerm,lastLogIndex) raft_log_ok freshness gate INSIDE the raft library itself, rather than the dispatcher applying freshness before the call as design.md originally specified -- this is the G6 vote-freshness closure from the later raft-commit-safety spec (own boundary, own validation), layered onto this spec's foundation; functionally strictly stronger, not a regression, and does not change any of this spec's own requirements or invariants. research.md documents one further non-blocking, functionally-equivalent routing drift (PreVote replies route via peer_ctrl_msg_handler's role+pre_vote_in_progress state check rather than a dedicated cluster.c OPCode_preVoteReply switch case as task 5.1 literally phrased it) -- already flagged in that file, not re-litigated here.\n\nMECHANICAL: Fresh ASAN build (nix develop + xce-build.sh controller asan) -- zero compiler/linker errors, binaries confirmed with session-fresh mtimes. Both authoritative standalone binaries run directly with ASAN_OPTIONS=detect_leaks=0 (per this project's convention, authoritative over meson's aggregate line): xcelib_test (raft library) 293 Tests, 0 Failures, 0 Ignored, OK, exit 0 -- includes 24 dedicated pre_vote-tagged RUN_TEST entries in test_raft.c (decision/tally purity, quorum WON/LOST/PENDING, duplicate-reply idempotency, out-of-range-peer guard, 5-node quorum, and 4 explicit invariant assertions incl. test_invariant_prevote_sequence_pure_no_persist and test_invariant_prevote_won_grants_no_leadership). controller_unit_test 972 Tests, 0 Failures, 0 Ignored, OK, exit 0 -- includes 28 dedicated PreVote-tagged tests across test_ha_prevote.c (12: start/no-mutation, majority-grant, no-majority-rearm, round-timeout-as-loss, reusable-after-loss, pending-noop, reply-over-real-transport, frozen-rejoin no-op x2, genuine-failover, heartbeat-mid-round-cancel, lost-then-later-elects), test_ha_failover_prevote.c (6: trigger rewire rdhup/liveness->PreVote not election, heartbeat-cancel, WON->real-election, config-less-node guard), test_ha_vote_gate.c (8: primary-role-denies, standby-fresh-heartbeat-denies, catch-up-grants-when-primary-unreachable, catch-up-denies-while-primary-live, stale-freshness-denies, stale-term-denies-no-adoption, granted-when-stickiness-lapsed, dead-primary-grants-despite-fresh-activity -- this last one is bug-660's own regression test), test_ha_role.c (1: dispatch-gate admits PreVote in voter roles), and test_ha_elect_ctx_lifecycle.c (1: 20 PreVote rounds x 2 votes with no carrier leak -- the sibling ha-elect-ctx-lifecycle spec's own regression test, confirming its refcounted-carrier fix holds under repeated PreVote churn). Zero :FAIL anywhere in either log, zero ASAN/crash signatures (AddressSanitizer, heap-buffer-overflow, use-after-free, double-free, SEGV) in either log. TBD/TODO/FIXME/HACK/XXX grep on the PreVote-boundary files (raft.c, raft.h, ha.c, ha.h, cluster.c, controller.capnp, test_ha_prevote.c, test_ha_failover_prevote.c, test_raft.c, test_ha_vote_gate.c): zero PreVote-related matches. Secrets grep: clean.\n\nRUNTIME LIVENESS (the blocking check, given the bug-1447 sibling finding and this spec's own live-election subject matter -- PreVote specifically governs election behavior under partition/unreachability): ran the canonical /xce-test-ha SEQUENTIAL 10-round soak (fresh session binaries) against a real ext4 deploy dir (~/tmp/xce-ha-prevote-soak, confirmed ext4 via df -T before the run, cleaned up after). Result: PASS -- 10/10 rounds, 33 jobs submitted, zero committed-job loss, zero walcompare divergence at any round boundary, all 3 nodes (ctrl-1, ctrl-2, ctrl-3) served as primary at least once (leadership-coverage maneuver satisfied by round 4). Term progressed 2->13 over 10 genuine kills -- climbing by exactly +1 per round in 9 of 10 rounds (clean single-jump elections, no runaway multi-round election-storm churn, exactly the disruption-suppression behavior this spec exists to produce) and +2 in exactly one round (round 9, term 10->12), coinciding with the log's single live PreVote DENIED event (candidate=ctrl-2 term=8 reason=freshness) -- a legitimate freshness-gate denial of a behind candidate, not a fault. Live PreVote activity confirmed directly from the per-node logs (not just inferred from job-loss absence): 13 PreVote round WON events across the three nodes' logs (6+5+2), 1 explicit PreVote DENIED (reason=freshness) log line carrying candidate id + reason exactly as Req 7.2 requires, and 2 'primary heartbeat arrived mid-PreVote -- cancelling the round' events (Req 4.2's mid-round-cancel path) -- meaning this run incidentally live-exercised the frozen-then-resumed re-attach path in addition to the genuine-failover path the sequential kill schedule directly drives, without needing a dedicated SIGSTOP-based scenario (design.md's Testing Strategy already covers the frozen-then-resumed case at the controller integration-test level via test_ha_prevote.c/test_ha_failover_prevote.c, run above; it does not call for a live SIGSTOP soak scenario, and none exists in run-ha-soak.sh's mode set -- confirmed by inspection). Explicitly grepped the full soak driver log AND all three per-node controller logs (11268/12012/11515 lines) for bug-1447's exact signature (AddressSanitizer, DEADLYSIGNAL, SEGV, ssl3_read_bytes, handle_ssl_handshake, io_worker): zero matches anywhere, including at round 9 -- the exact round bug-1447 hit in raft-drive-async's own soak. Also grepped broadly for any other crash/abort signature (Segmentation fault, core dumped, Aborted, double free, heap-buffer-overflow, use-after-free): zero matches. This does not prove bug-1447 is fixed or globally absent (it is a rare, timing-dependent race in raft-drive-async's own reconnect-offload cadence, a different subsystem this spec's boundary does not touch) -- but it does establish that this spec's own PreVote-gated election path runs clean through a full 10-round supported failover lifecycle with live PreVote activity on every round, satisfying this spec's own regression contract (Req 5.4, 4.x).\n\nOWNERSHIP: bug-1447 remains raft-drive-async/netio.c territory (see that spec's own NO-GO record, bug-1447 in buglog.json); noted here only as a risk actively checked for and ruled out in this run, not adopted as this spec's blocker. No new bugs found or fixed during this validation pass -- the tree's PreVote-adjacent bug history (605, 606, 660, 1421, 1424, 1430, 1435) was reviewed and every fix confirmed present and correct in the current source (e.g. bug-1430's CAS-based UAF fix in ha_prevote_round_timeout_fn read directly from ha.c:7216-7240).",
    "known_limitations": [
      "bug-1447 (open, NOT-FIXED, owned by raft-drive-async / src/lib/netio.c, not this spec): a timing-dependent ASAN SEGV in handle_ssl_handshake/io_worker triggered by raft-drive-async's own reconnect-offload cadence. Did not reproduce in this spec's clean 10/10 run (including at round 9, the exact round it hit for raft-drive-async), but as a shared-codebase race condition it cannot be ruled out for all future runs by one clean pass. No action item for ha-prevote; tracked entirely under raft-drive-async's remediation.",
      "The deliberate per-round ha_elect_ctx_t carrier leak that this spec's own tasks.md Implementation Notes (task 4.2) had explicitly deferred to a follow-up spec is now resolved in the current tree by ha-elect-ctx-lifecycle (commit 6529dcf, reference-counted carrier via atomic_exchange + release-on-last-ref in ha_start_prevote_async) -- confirmed present in the current ha.c source and covered by a live-passing regression test (test_many_prevote_rounds_no_carrier_leak, 20 PreVote rounds x 2 votes, no leak). That sibling spec's own spec.json is still phase=tasks-generated pending its own validate-impl pass (scheduled to run after this one in the same worktree); this note records that ha-prevote's own known-limitation is already satisfied in-tree regardless of that sibling's administrative validation status.",
      "design.md's original Service Interface signature for raft_on_pre_vote_request (2 params: from_idx, cand_term) is now stale documentation -- the shipped function additionally takes cand_last_log_term/cand_last_log_index and runs the (lastLogTerm,lastLogIndex) freshness gate inside the raft library itself (the later raft-commit-safety spec's G6 vote-freshness closure, layered onto this spec's foundation). Functionally strictly stronger than the original design, not a regression; a future doc pass could update design.md's Service Interface block to match the shipped signature.",
      "research.md's own file already documents one non-blocking, functionally-equivalent routing drift: PreVote replies are routed via peer_ctrl_msg_handler's role+pre_vote_in_progress state check rather than a dedicated cluster.c OPCode_preVoteReply switch case as task 5.1's text literally phrased it. Not re-verified independently in this pass beyond confirming the request-side opcode (preVoteReply @39) is still emitted and the reply still reaches the tally live (confirmed via the soak's 13 live PreVote round WON events)."
    ]
  }
}