myxce/.kiro/specs/raft-library/requirements.md

Requirements Document

Introduction

XCE controller developers need a correct, independently-testable election consensus library. The existing election logic in ha.c is tightly coupled to the iomgr/SSL/capnp transport layer, preventing deterministic unit testing without a live cluster. This feature extracts the pure Raft-style election state machine into a self-contained C library (libxce) with callback interfaces, so election correctness can be verified with a deterministic simulator and then wired into the controller via thin adapters.

Boundary Context

• In scope: Election state machine (term, voted_for, role, quorum evaluation), vote request/reply processing, leader-lost failover trigger, callback interface definition, deterministic simulator test harness.
• Out of scope: Network transport (SSL, iomgr, capnp), WAL truncation on failover, config replication push, cluster_init_phase2, persistence to disk (remains in caller).
• Adjacent expectations: The library fires callbacks when it needs to send a vote RPC or when role transitions occur; the controller adapter implements those callbacks using the existing iomgr/SSL transport. Disk persistence of term/voted_for remains the caller's responsibility via an on_persist callback.

Requirements

1. Election State

Objective: As a controller developer, I want the election state machine to be self-contained, so that I can test Raft invariants without network infrastructure.

Acceptance Criteria

1. The Raft Library shall maintain per-node state: current term (uint64_t), role (FOLLOWER/CANDIDATE/LEADER), voted_for candidate index (or -1), and n_peers count.
2. When initialized, the Raft Library shall set term=0, role=FOLLOWER, voted_for=-1.
3. The Raft Library shall expose no dependency on endpoint_t, SSL_CTX, iomgr_t, cluster_t, or capnp types in its public header.

2. Election Start

Objective: As a controller developer, I want to start an election by calling a single function, so that the library handles the Raft protocol steps.

Acceptance Criteria

1. When raft_start_election() is called, the Raft Library shall increment term, record self-vote (voted_for = self index), set role to CANDIDATE, and invoke the on_send_vote_request callback for each peer.
2. When raft_start_election() is called with n_peers == 1 (single-node), the Raft Library shall skip peer RPCs and immediately transition to LEADER via the on_become_leader callback.
3. If raft_start_election() is called while role is already CANDIDATE or LEADER, the Raft Library shall return an error without modifying state.

3. Vote Request Handling

Objective: As a controller developer, I want incoming vote requests to be evaluated correctly, so that only one candidate wins per term.

Acceptance Criteria

1. When raft_on_vote_request() is called with term > current_term, the Raft Library shall update current_term, clear voted_for, set role to FOLLOWER, and evaluate the request.
2. When raft_on_vote_request() is called with term < current_term, the Raft Library shall deny the vote and return the current term to the caller.
3. When raft_on_vote_request() is called with term == current_term and voted_for is already set to a different candidate, the Raft Library shall deny the vote (one vote per term invariant).
4. When raft_on_vote_request() is called and the candidate passes the freshness gate (candidate config_tag ≥ local config_tag AND candidate log LSN ≥ local applied LSN), the Raft Library shall grant the vote, record voted_for = candidate index, and invoke the on_persist callback.
5. While role is LEADER or CANDIDATE_COMMITTING, the Raft Library shall deny all incoming vote requests regardless of term.

4. Vote Reply Handling and Quorum

Objective: As a controller developer, I want vote replies to be accumulated correctly so that the library detects quorum and reports the election result.

Acceptance Criteria

1. When raft_on_vote_reply() is called with a granted vote and votes_granted reaches quorum (ceil((n_peers+1)/2)), the Raft Library shall set role to LEADER and invoke the on_become_leader callback exactly once.
2. When raft_on_vote_reply() is called with term > current_term, the Raft Library shall update current_term, set role to FOLLOWER, and invoke on_become_follower.
3. When raft_on_vote_reply() is called after the election is already decided (on_become_leader or on_become_follower already fired), the Raft Library shall be a no-op.
4. When all peers have replied and votes_granted < quorum, the Raft Library shall set role to FOLLOWER and invoke on_become_follower.

5. Leader Lost (RDHUP Failover)

Objective: As a controller developer, I want to notify the library when the primary control channel is lost, so that a new election is triggered correctly.

Acceptance Criteria

1. When raft_on_leader_lost() is called, the Raft Library shall clear voted_for, set role to FOLLOWER, and invoke the on_start_election_after_delay callback with the priority-based delay (priority_index * election_timeout_ms).
2. When the delay elapses and voted_for is set to a non-self candidate for the new term (another node won during the delay), the Raft Library shall skip starting its own election.
3. When the delay elapses and voted_for is unset or set to self, the Raft Library shall invoke raft_start_election() internally.

6. Election Timeout

Objective: As a controller developer, I want an election timeout to fire if no leader is elected within the configured window, so that the cluster does not stall.

Acceptance Criteria

1. When raft_start_election() is called, the Raft Library shall invoke the on_schedule_timeout callback with election_timeout_ms.
2. When the timeout fires (caller invokes raft_on_election_timeout()) and the election is not yet decided, the Raft Library shall invoke on_become_follower and cancel any pending vote tracking.
3. When the timeout fires and the election is already decided, the Raft Library shall invoke on_cancel_timeout and be a no-op.

7. Callback Interface

Objective: As a controller developer, I want the library to communicate all side effects through callbacks, so that I can inject a deterministic test double without network I/O.

Acceptance Criteria

1. The Raft Library shall define a raft_callbacks_t struct with the following function pointer fields: on_send_vote_request, on_become_leader, on_become_follower, on_persist (term/voted_for), on_schedule_timeout, on_cancel_timeout.
2. The Raft Library shall accept a user-supplied void* context pointer at init time and pass it unchanged to every callback invocation.
3. If any required callback pointer is NULL at raft_init() time, the Raft Library shall return a non-zero error code.

8. Deterministic Test Harness

Objective: As a controller developer, I want a simulator that exercises the Raft Library without network I/O, so that I can verify correctness under controlled timing.

Acceptance Criteria

1. The Raft Library test suite shall include a deterministic N-node simulator that routes on_send_vote_request callbacks to the target node's raft_on_vote_request/reply functions synchronously, without real network I/O.
2. The test suite shall verify the following invariants: (a) exactly one LEADER per election in a 3-node cluster with no failures (no split-brain), (b) a second vote request in the same term is denied, (c) a single-node cluster self-elects immediately, (d) election timeout causes re-election.
3. The Raft Library shall compile and all tests shall pass under ASAN with ASAN_OPTIONS=detect_leaks=0.

9. Integration with Controller

Objective: As a controller developer, I want a thin adapter in ha.c that connects the Raft Library to the existing iomgr/SSL transport, so that ha.c no longer owns election state directly.

Acceptance Criteria

1. The adapter shall implement all raft_callbacks_t callbacks using existing endpoint_t and iomgr transport without duplicating election state management.
2. After integration, ha_start_election_async shall delegate term management, voted_for tracking, and quorum evaluation to the Raft Library; it shall no longer directly write ha->term or ha->voted_for fields outside of the adapter callbacks.
3. The existing XCE_TESTING hook seams (ha_phase2_hook, ha_rdhup_election_hook) shall remain functional after integration so that existing test cases continue to pass.