myxce/.kiro/specs/raft-library/design.md

Design Document: raft-library

Overview

This feature extracts the Raft-style election state machine from ha.c into a self-contained C library (src/lib/raft/) that is decoupled from the iomgr/SSL/capnp transport layer. The library exposes a pure state machine with a callback interface for all transport effects, enabling deterministic unit testing without a live cluster.

Purpose: Provide a correct, independently verifiable Raft election implementation that eliminates split-brain bugs traceable to entangled state transitions in ha.c.

Users: Controller developers will use this to wire the election protocol into the controller via a thin adapter, and to verify protocol correctness via a deterministic N-node simulator.

Impact: ha.c becomes an adapter over raft.h rather than the owner of election state. ha_state_t.term, ha_state_t.voted_for_*, and quorum logic move to raft_state_t.

Goals

• Correct Raft invariants: one vote per term, quorum safety, no split-brain in 3-node deterministic tests
• Transport-free public API: raft.h has zero dependency on endpoint_t, SSL_CTX, iomgr_t, cluster_t
• Exhaustive protocol-level test cases covering all invariants specified in requirements
• Drop-in wiring: the controller adapter preserves existing XCE_TESTING hook seams

Non-Goals

• Full Raft log replication (XCE is single-writer; only leader election is needed)
• WAL truncation on failover (remains in ha.c/persist.c)
• Config replication push (remains in config_repl.c)
• Disk persistence of term/voted_for (caller's responsibility via on_persist callback)
• Multi-Raft or multi-group election

---

Boundary Commitments

This Spec Owns

• src/lib/raft/raft.h — public API: raft_state_t, raft_callbacks_t, all function signatures
• src/lib/raft/raft.c — pure state machine implementation
• src/lib/raft/raft_sim.c — deterministic N-node simulator (test harness only)
• src/lib/testsuite/test_raft.c — protocol-level test suite (≥ 20 test cases)
• src/lib/meson.build additions for raft.c and test_raft.c
• The definition of raft_callbacks_t and the semantic contract for each callback

Out of Boundary

• ha.c adapter wiring (separate task, touches ha.c but does not own raft.h)
• SSL/iomgr transport implementation of callbacks
• WAL truncation, config push, cluster_init_phase2
• Persistence to disk (ha-term.bin) — caller implements via on_persist callback
• Controller startup sequencing (myapp_run)

Allowed Dependencies

• raft.h and raft.c: stdint.h, string.h, errno.h only — no pthread, no OS primitives
• The library has zero threading dependencies; concurrency is entirely the caller's responsibility
• No libxce dependencies from raft.h (headers that include endpoint_t or similar are forbidden)
• Test file may include <unity/unity.h> (project test framework) and <pthread.h> (for test harness only)
• raft_sim.c may include <string.h> and <stdio.h> for the simulator and dump log

Revalidation Triggers

• Changes to raft_callbacks_t shape (adding/removing fields) require re-check of ha.c adapter
• Changes to quorum formula require re-running protocol-level tests
• Changes to role enum values require scanning ha.c for role-comparison code

---

Architecture

Existing Architecture Analysis

ha.c currently mixes three concerns:

1. Raft state — term, voted_for, role, quorum tracking (ha_elect_ctx_t)
2. Transport — endpoint_connect(), iomgr_monitor(), SSL setup, capnp encoding/decoding
3. Post-election actions — cluster_init_phase2(), push_config_to_standbys(), WAL truncation

The split-brain bugs arise from concern #1 being modified across threads without a consistent lock discipline: the transport layer (concern #2) and post-election hooks (concern #3) call into Raft state functions without a clear ownership model.

Architecture Pattern

graph TB
    subgraph libxce_raft
        RaftState[raft_state_t]
        RaftAPI[raft.c Pure SM]
        RaftSim[raft_sim.c N-node sim]
    end

    subgraph ha_adapter
        HaAdapter[ha.c adapter]
    end

    subgraph transport
        Iomgr[iomgr endpoint_t SSL]
        Persist[ha_persist_term]
        DelayMgr[delaymgr timeout]
    end

    HaAdapter -->|calls| RaftAPI
    RaftAPI -->|callbacks| HaAdapter
    HaAdapter -->|drives| Iomgr
    HaAdapter -->|drives| Persist
    HaAdapter -->|drives| DelayMgr
    RaftSim -->|calls| RaftAPI

Key decisions:

• No threads in the library: raft.c is fully synchronous. The caller holds a lock before calling any raft_* function.
• Callbacks fire synchronously: The library calls back into the adapter within the same call frame. The adapter is responsible for not holding library locks during slow I/O.
• Role enum is self-contained: raft_role_t in raft.h is independent of ctrl_role_t in ha.h; the adapter maps between them.

Technology Stack

Layer	Choice	Role
Language	C gnu17	Matches project standard
OS dependencies	None (stdint.h, string.h, errno.h only)	Library is OS-free; concurrency is entirely caller-owned
Threading	Caller-owned (pthread or none)	raft_state_t is not thread-safe by itself; caller holds whatever lock fits their threading model
Atomics	None in library	Caller uses atomics for ha_state_t integration if needed
Test framework	Unity (existing)	Consistent with controller testsuite
Build	Meson (existing)	static_library('raft', 'raft.c') added to src/lib/meson.build

---

File Structure Plan

src/lib/
├── raft/
│   ├── raft.h          # Public API: raft_state_t, raft_callbacks_t, all function signatures
│   ├── raft.c          # Pure state machine: term, voted_for, role, quorum evaluation
│   └── raft_sim.c      # N-node deterministic simulator (compiled only under XCE_TESTING)
├── testsuite/
│   └── test_raft.c     # Protocol-level test suite (≥ 20 cases)
└── meson.build         # Add raft.c to xcelib; add test_raft.c to xcelib_test

Modified files:

• src/lib/meson.build — add raft/raft.c to xcelib sources; add testsuite/test_raft.c
• src/controller/ha.h — add #include "raft/raft.h" and raft_state_t raft field to ha_state_t (adapter task)
• src/controller/ha.c — delegate election logic to raft_* API (adapter task, separate from library task)

---

System Flows

Election State Machine

stateDiagram-v2
    [*] --> FOLLOWER : raft_init
    FOLLOWER --> CANDIDATE : raft_start_election
    CANDIDATE --> LEADER : quorum reached
    CANDIDATE --> FOLLOWER : timeout or higher term
    LEADER --> FOLLOWER : higher term seen
    FOLLOWER --> FOLLOWER : raft_on_leader_lost clears voted_for
    FOLLOWER --> CANDIDATE : raft_on_election_timeout

Vote Request Processing

sequenceDiagram
    participant Caller
    participant RaftSM as raft.c
    participant CB as callbacks

    Caller->>RaftSM: raft_on_vote_request(state, from, term)
    alt term < current_term
        RaftSM-->>Caller: RAFT_DENY stale term
    else term > current_term
        RaftSM->>RaftSM: update term, clear voted_for, role=FOLLOWER
    end
    alt already voted for different candidate same term
        RaftSM-->>Caller: RAFT_DENY already voted
    else on_vote_eligible returns 0
        RaftSM->>CB: on_vote_eligible(ctx, from, term)
        CB-->>RaftSM: 0 deny
        RaftSM-->>Caller: RAFT_DENY policy
    else all checks pass
        RaftSM->>CB: on_vote_eligible(ctx, from, term)
        CB-->>RaftSM: 1 grant
        RaftSM->>CB: on_persist(term, from)
        RaftSM->>RaftSM: voted_for = from
        RaftSM-->>Caller: RAFT_GRANT
    end

on_vote_eligible contract (implemented by ha.c adapter):

• Return 0 if local role is LEADER or CTRL_PRIMARY_COMMITTING
• Return 0 if STANDBY with fresh heartbeat (primary still alive)
• Return 0 if in-catch-up state
• Return 0 if candidate config_tag < local config_tag, or equal tag with lower LSN
• Return 1 otherwise (candidate is eligible)

RDHUP Failover (Priority Jitter)

sequenceDiagram
    participant Caller
    participant RaftSM as raft.c
    participant CB as callbacks

    Caller->>RaftSM: raft_on_leader_lost(state, my_priority)
    RaftSM->>RaftSM: clear voted_for, role=FOLLOWER
    RaftSM->>CB: on_start_election_after_delay(priority * election_timeout_ms)
    note over Caller: delay elapses in caller's timer
    Caller->>RaftSM: raft_check_and_start_election(state)
    alt voted_for set to non-self (received vote during delay)
        RaftSM-->>Caller: RAFT_SKIP_ELECTION
    else
        RaftSM->>RaftSM: raft_start_election internally
        RaftSM->>CB: on_send_vote_request per peer
    end

---

Components and Interfaces

Component Summary

Component	Layer	Intent	Req Coverage	Contracts
raft_state_t	Data	Raft node state	1.1, 1.2	State
raft_callbacks_t	Interface	Transport/effect boundary	7.1, 7.2, 7.3	Service
raft.c State Machine	Logic	Pure election logic	1–6	Service
raft_sim.c Simulator	Test	Deterministic N-node harness	8.1, 8.2	Service
ha.c Adapter	Integration	Wire raft.c to iomgr/SSL	9.1, 9.2, 9.3	Service

---

Core Data Structures

/* raft_role_t: self-contained role enum (does not depend on ctrl_role_t) */
typedef enum {
    RAFT_FOLLOWER   = 0,
    RAFT_CANDIDATE  = 1,
    RAFT_LEADER     = 2,
} raft_role_t;

/* raft_state_t: pure Raft consensus state for one node.
 *
 * Contains only the Raft invariant state.  Application-specific fields
 * (config_tag, log_lsn, heartbeat timestamps, ctrl_role) stay in ha_state_t;
 * they are accessed by the caller via the on_vote_eligible callback. */
typedef struct {
    uint64_t      term;             /* current term (monotonically increasing) */
    raft_role_t   role;             /* current role */
    int           voted_for;        /* peer index voted for this term; -1=none; self_idx=self-vote */
    uint64_t      voted_for_term;   /* term in which voted_for was set */
    int           n_peers;          /* total cluster size (including self) */
    int           self_idx;         /* this node's index in [0, n_peers) */
    int           votes_granted;    /* votes collected in current election */
    int           election_decided; /* 1 once on_become_leader/follower fired */
    uint32_t      election_timeout_ms;
    int           my_priority;      /* priority index for jitter (0 = highest) */
} raft_state_t;

/* raft_callbacks_t: all side effects the library fires.
 *
 * The library owns Raft invariants (term ordering, one-vote-per-term, quorum).
 * All application-specific policy (freshness gate, role gate, catch-up state,
 * heartbeat liveness) is delegated to on_vote_eligible so the library is
 * fully independent of XCE domain logic.
 */
typedef struct {
    /* Send a RequestVote RPC to peer peer_idx for the given term.
     * The caller encodes config_tag, log_lsn, and any other fields in the wire RPC. */
    int  (*on_send_vote_request)(void *ctx, int peer_idx, uint64_t term);

    /* Application-specific vote eligibility gate.
     * Called after Raft invariants pass (correct term, voted_for unset).
     * Returns 1 to grant, 0 to deny.  Receives the candidate's peer_idx and
     * the incoming term so the caller can check freshness (config_tag, log_lsn),
     * role gate (LEADER/COMMITTING deny), catch-up status, heartbeat liveness.
     * The library grants the vote only if this callback returns 1. */
    int  (*on_vote_eligible)(void *ctx, int from_idx, uint64_t term);

    /* Node has won the election */
    void (*on_become_leader)(void *ctx, uint64_t term);
    /* Node lost election or saw higher term; leader_idx = -1 if unknown */
    void (*on_become_follower)(void *ctx, uint64_t term, int leader_idx);
    /* Persist term + voted_for before granting vote (durability guarantee) */
    int  (*on_persist)(void *ctx, uint64_t term, int voted_for);
    /* Schedule election timeout (fires raft_on_election_timeout after delay_ms) */
    void (*on_schedule_timeout)(void *ctx, uint32_t delay_ms);
    /* Cancel pending election timeout */
    void (*on_cancel_timeout)(void *ctx);
    /* Request delayed election start after RDHUP (priority-based jitter) */
    void (*on_start_election_after_delay)(void *ctx, uint32_t delay_ms);
    /* Trace callback for observability (optional, may be NULL) */
    void (*on_trace)(void *ctx, const char *event, const char *detail);
} raft_callbacks_t;

---

Public API (raft.h)

/*
 * raft_config_t: initialization context for a Raft node.
 *
 * Encapsulates all configuration so raft_init() has a single, extensible
 * parameter rather than a flat list.  Matches the existing XCE pattern
 * (ha_config_t, persist_config_t).  Zero-initialize and set only the
 * fields you need — the library fills in safe defaults for unset fields.
 *
 * Persistence model:
 *   initial_term / initial_voted_for are restored by the caller from disk
 *   before raft_init().  The library never reads storage; it only writes
 *   via on_persist callbacks.  Pass {0, -1} on first boot.
 */
typedef struct {
    int           n_peers;              /* total cluster size (including self) */
    int           self_idx;             /* this node's index in [0, n_peers) */
    uint32_t      election_timeout_ms;  /* base election timeout; 0 → default 2000ms */
    int           my_priority;          /* priority index for RDHUP jitter (0 = highest) */
    uint64_t      initial_term;         /* term restored from disk; 0 on first boot */
    int           initial_voted_for;    /* voted_for restored from disk; -1 if none */
    const raft_callbacks_t *cbs;        /* all callbacks; required callbacks must be non-NULL */
    void         *ctx;                  /* opaque caller context passed to every callback */
} raft_config_t;

/*
 * raft_init: initialize raft_state_t from a raft_config_t.
 * on_vote_eligible and on_trace may be NULL (optional policy / observability).
 * All other callbacks in cbs must be non-NULL.
 * Returns 0 on success; -EINVAL if a required callback is NULL or n_peers < 1.
 */
int raft_init(raft_state_t *s, const raft_config_t *cfg);

/*
 * raft_start_election: begin a new election.
 * Increments term, records self-vote (voted_for=self_idx), transitions to
 * CANDIDATE, fires on_send_vote_request for each peer, fires on_schedule_timeout.
 * Returns 0 on success; -EALREADY if already CANDIDATE/LEADER;
 * RAFT_ELECTED_IMMEDIATELY for single-node (n_peers==1).
 */
int raft_start_election(raft_state_t *s);

/*
 * raft_on_vote_request: process an incoming RequestVote RPC.
 * Applies Raft invariants (term comparison, one-vote-per-term).
 * If invariants pass, calls on_vote_eligible to allow application policy.
 * Fires on_persist before granting.
 * Returns RAFT_GRANT (1) or RAFT_DENY (0); sets *reply_term to current term.
 */
int raft_on_vote_request(raft_state_t *s, int from_idx, uint64_t term,
                          uint64_t *reply_term);

/*
 * raft_on_vote_reply: process a reply to our RequestVote.
 * Returns RAFT_BECAME_LEADER, RAFT_BECAME_FOLLOWER, or RAFT_PENDING.
 */
int raft_on_vote_reply(raft_state_t *s, int from_idx, int granted, uint64_t term);

/*
 * raft_on_election_timeout: called by the timer when no quorum was reached.
 * Transitions CANDIDATE → FOLLOWER; fires on_become_follower, on_cancel_timeout.
 */
void raft_on_election_timeout(raft_state_t *s);

/*
 * raft_on_leader_lost: primary control channel dropped (RDHUP).
 * Clears voted_for, transitions to FOLLOWER, fires on_start_election_after_delay.
 */
void raft_on_leader_lost(raft_state_t *s);

/*
 * raft_check_and_start_election: call after the delay from raft_on_leader_lost.
 * Starts election only if we have not already voted for a non-self candidate
 * in the new term (i.e., another node won during the jitter window).
 * Returns RAFT_SKIP_ELECTION or RAFT_ELECTION_STARTED.
 */
int raft_check_and_start_election(raft_state_t *s);

/* Return codes */
#define RAFT_OK                  0
#define RAFT_GRANT               1
#define RAFT_DENY                0
#define RAFT_BECAME_LEADER       1
#define RAFT_BECAME_FOLLOWER     2
#define RAFT_PENDING             3
#define RAFT_ELECTED_IMMEDIATELY 4
#define RAFT_SKIP_ELECTION       5
#define RAFT_ELECTION_STARTED    6

Preconditions for all functions: Caller must hold a mutex protecting raft_state_t. Postconditions: Callbacks may fire synchronously within the call; caller must not hold transport locks during callback execution.

---

Deterministic Simulator (raft_sim.c)

The simulator replaces real network I/O with synchronous in-process callback routing.

/* raft_sim_node_t: one simulated cluster node */
typedef struct {
    raft_state_t   state;
    raft_callbacks_t cbs;
    void          *ctx;   /* points back to sim_cluster_t */
    int            idx;
    int            partition; /* 1 = isolated from other nodes */
} raft_sim_node_t;

/* raft_sim_cluster_t: N-node simulation cluster */
typedef struct {
    raft_sim_node_t nodes[RAFT_SIM_MAX_NODES];
    int             n;
    int             leader_count; /* incremented on each on_become_leader */
} raft_sim_cluster_t;

int  raft_sim_init(raft_sim_cluster_t *c, int n_nodes, uint32_t election_timeout_ms);
void raft_sim_start_all_elections(raft_sim_cluster_t *c); /* triggers all nodes simultaneously */
void raft_sim_deliver_pending(raft_sim_cluster_t *c);     /* routes buffered RPCs */
void raft_sim_fire_timeout(raft_sim_cluster_t *c, int node_idx);
void raft_sim_partition(raft_sim_cluster_t *c, int node_idx, int isolated);

---

Protocol-Level Use Cases (Testing Strategy)

The following test cases verify Raft invariants at the protocol level, independent of transport. Each test uses raft_sim_cluster_t and runs to completion synchronously.

Group A: Basic Election Correctness (Req 1–4, 7, 8)

TC	Scenario	Nodes	Expected
A1	Single-node cluster starts election	1	Immediately LEADER, on_become_leader fires once
A2	3-node simultaneous election, node 0 has priority	3	Exactly 1 LEADER; nodes 1,2 grant to node 0 and become FOLLOWER
A3	3-node election, node 0 wins, node 1 starts late election same term	3	Node 1's election denied (voted_for set); still 1 LEADER
A4	5-node election, quorum = 3	5	Leader wins after 3 grants; 2 remaining nodes become FOLLOWER
A5	Node receives vote request in LEADER role	3	Denies regardless of term; role stays LEADER

Group B: Voted_For Invariant (Req 3, one-vote-per-term)

TC	Scenario	Expected
B1	Node A and B both request vote from node C in same term	C grants only the first requester; second is denied
B2	Node sees vote request with term > current_term	Updates term, clears voted_for, re-evaluates request
B3	Node sees vote request with term < current_term	Denies immediately, returns current term
B4	Node already voted for X in term T; receives request from Y in term T	Denies Y; voted_for stays X
B5	Node granted vote for X in term T; receives request from X again in term T	Grants (idempotent re-grant to same candidate)

Group C: on_vote_eligible Policy Callback (Req 3.4, 3.5)

The library calls on_vote_eligible after Raft invariants pass. These tests use a mock on_vote_eligible that returns a configurable value:

TC	Scenario	on_vote_eligible returns	Expected
C1	All Raft invariants pass; on_vote_eligible returns 0	0	Denied (policy)
C2	All Raft invariants pass; on_vote_eligible returns 1	1	Granted
C3	on_vote_eligible is NULL at init	—	All eligible candidates granted (no policy gate)
C4	Raft invariant fails (stale term); on_vote_eligible NOT called	N/A	Denied before callback fires
C5	Voted-for already set; on_vote_eligible NOT called	N/A	Denied before callback fires

Note: The freshness gate (config_tag, log_lsn), role gate (LEADER/COMMITTING deny), and heartbeat-freshness gate are ha.c adapter responsibilities implemented inside on_vote_eligible — not tested in the library test suite.

Group H: Persistence and Restart (Req 1.2, on_persist)

TC	Scenario	Expected
H1	raft_init with cfg.initial_term=5, cfg.initial_voted_for=1	state.term=5, state.voted_for=1 immediately
H2	raft_init with cfg.initial_term=0, cfg.initial_voted_for=-1 (first boot)	state.term=0, state.voted_for=-1
H3	Vote granted; on_persist fires before voted_for is recorded	on_persist called with (term, from_idx) before grant returns
H4	on_persist returns non-zero (disk failure)	Vote denied; voted_for unchanged
H5	raft_start_election: on_persist fires for self-vote	on_persist called with (new_term, self_idx)

Group D: Quorum and Result (Req 4)

TC	Scenario	Expected
D1	3-node: 1 grant received (self + 1 = 2 = quorum)	LEADER immediately
D2	3-node: both peers deny	election_decided=1, FOLLOWER, on_become_follower fires
D3	on_become_leader fires: second vote reply arrives	No-op (election_decided guard)
D4	Reply with term > current_term	Update term, FOLLOWER, on_become_follower fires

Group E: Election Timeout (Req 6)

TC	Scenario	Expected
E1	Timeout fires before quorum	on_become_follower; on_cancel_timeout
E2	Timeout fires after quorum	No-op (election already decided)
E3	New election after timeout	term increments; on_schedule_timeout fires again

Group F: RDHUP Failover (Req 5)

TC	Scenario	Expected
F1	3-node: node 0 killed; nodes 1,2 call raft_on_leader_lost	Node 1 (priority=1) waits 1×timeout; node 2 (priority=2) waits 2×timeout
F2	During node 1's delay, node 1 receives vote request from another election starter	voted_for set; raft_check_and_start_election returns RAFT_SKIP_ELECTION
F3	During node 1's delay, no vote request arrives	raft_check_and_start_election starts election; 1 LEADER
F4	Simultaneous RDHUP (all remaining nodes fire at same time, no delay difference)	Only one LEADER (self-vote prevents double grant)

Group G: Split-Brain Prevention (Req 4, overall correctness)

TC	Scenario	Expected
G1	3-node simultaneous elections all terms (100 random election rounds via simulator)	Never more than 1 LEADER per term across all rounds
G2	Network partition: nodes 0+1 isolated from node 2	Partition of 2 can elect; partition of 1 cannot

---

Requirements Traceability

Req	Summary	Component	Interface/Contract
1.1	State fields	raft_state_t	Data struct
1.2	Init to FOLLOWER	raft_init	Function
1.3	No transport types in header	raft.h boundary	Boundary commit
2.1	Election: incr term, self-vote, send RPCs	raft_start_election	Function
2.2	Single-node shortcircuit	raft_start_election	RAFT_ELECTED_IMMEDIATELY
2.3	Error if already CANDIDATE/LEADER	raft_start_election	-EALREADY
3.1	Higher term → update, re-evaluate	raft_on_vote_request	Function
3.2	Lower term → deny	raft_on_vote_request	Return value
3.3	Same term, different voted_for → deny	raft_on_vote_request	voted_for check
3.4	Freshness gate → grant + persist	raft_on_vote_request	on_persist callback
3.5 (5)	LEADER/COMMITTING → deny	raft_on_vote_request	Role check
4.1	Quorum → LEADER	raft_on_vote_reply	RAFT_BECAME_LEADER
4.2	Higher term in reply → FOLLOWER	raft_on_vote_reply	RAFT_BECAME_FOLLOWER
4.3	Post-decision no-op	raft_on_vote_reply	election_decided guard
4.4	All replied, no quorum → FOLLOWER	raft_on_vote_reply	vote counting
5.1	Leader lost → clear voted_for + delay callback	raft_on_leader_lost	on_start_election_after_delay
5.2	Already voted during delay → skip	raft_check_and_start_election	RAFT_SKIP_ELECTION
5.3	Not voted → start election	raft_check_and_start_election	RAFT_ELECTION_STARTED
6.1	Start election → schedule timeout	raft_start_election	on_schedule_timeout
6.2	Timeout fires, undecided → follower	raft_on_election_timeout	on_become_follower
6.3	Timeout fires, decided → no-op	raft_on_election_timeout	election_decided guard
7.1	raft_callbacks_t struct definition	raft.h	Data struct
7.2	void* ctx threading	All callbacks	Function signatures
7.3	NULL callback → error	raft_init	-EINVAL
8.1	Deterministic simulator	raft_sim.c	Simulator API
8.2	4 invariant test cases	test_raft.c	Groups A–G
8.3	ASAN clean	Build config	Meson test suite
9.1	Adapter implements callbacks	ha.c adapter (separate task)	raft_callbacks_t impl
9.2	ha.c delegates state to library	ha.c adapter (separate task)	ha_state_t.raft field
9.3	XCE_TESTING hooks preserved	ha.c adapter (separate task)	Existing seams

---

Error Handling

Error	Source	Response
NULL callback at init	raft_init	Return -EINVAL before any state is set
raft_start_election called while CANDIDATE	State check	Return -EALREADY
on_persist callback returns error	raft_on_vote_request	Do not grant vote; return RAFT_DENY
on_send_vote_request fails for one peer	Election start	Count as no-reply for that peer; election proceeds with remaining peers

---

Observability and Debug Tracing

Protocol correctness requires that every state transition is observable without a live cluster. The library provides two tracing mechanisms:

1. Trace Callback (production + test)

Add an optional on_trace field to raft_callbacks_t:

/* on_trace: fired on every state transition or significant decision.
 * event: short string literal, e.g. "vote_granted", "became_leader", "deny_stale_term"
 * detail: formatted human-readable context string (caller-allocated)
 * May be NULL to suppress tracing. */
void (*on_trace)(void *ctx, const char *event, const char *detail);

Every call to a public raft_* function fires on_trace at entry and at each decision point:

Trace event	When fired
election_start	raft_start_election begins
self_vote	Self-vote recorded
vote_request_in	raft_on_vote_request called
vote_granted	Vote granted to candidate
vote_denied_stale_term	Denied: term < current
vote_denied_already_voted	Denied: voted_for already set
vote_denied_role	Denied: LEADER or COMMITTING
vote_denied_freshness	Denied: candidate not fresh
vote_reply_in	raft_on_vote_reply called
quorum_reached	Votes_granted >= quorum
became_leader	on_become_leader fired
became_follower	on_become_follower fired
election_timeout	raft_on_election_timeout called
leader_lost	raft_on_leader_lost called
jitter_skip	raft_check_and_start_election → SKIP
term_update	term incremented or bumped by peer

In production (ha.c adapter), on_trace routes to INFO(...) logging. In tests, on_trace appends to a per-node trace buffer; test assertions can inspect the event sequence to verify protocol correctness beyond role/term checks.

2. State Snapshot (raft_snapshot)

/* raft_snapshot_t: point-in-time read of all Raft state for logging/debugging */
typedef struct {
    uint64_t     term;
    raft_role_t  role;
    int          voted_for;
    uint64_t     voted_for_term;
    int          votes_granted;
    int          election_decided;
} raft_snapshot_t;

/* Fills snap with current state. Thread-safe (caller must hold lock). */
void raft_snapshot(const raft_state_t *s, raft_snapshot_t *snap);

The raft_snapshot function is used by:

• The simulator to print cluster state after each test step
• The adapter to log state on RDHUP for post-mortem debugging
• The ASAN test runner to dump state on test failure

3. Simulator Trace Output

raft_sim_cluster_t records a timestamped event log for each RPC and state transition. After each test, raft_sim_dump_log(cluster, FILE*) prints the protocol trace in a human-readable format, e.g.:

[step 0] node0: election_start term=2
[step 1] node0→node1: vote_request term=2 tag=0 lsn=0
[step 1] node0→node2: vote_request term=2 tag=0 lsn=0
[step 2] node1: vote_granted to node0 term=2
[step 2] node2: vote_granted to node0 term=2
[step 3] node0: quorum_reached votes=3 → became_leader term=2

This trace is printed to stderr on test failure and can be enabled unconditionally with RAFT_SIM_VERBOSE=1.

File structure additions

src/lib/raft/
├── raft.h          # + on_trace, raft_snapshot_t, raft_snapshot()
├── raft.c          # + trace emission at every decision point
└── raft_sim.c      # + raft_sim_dump_log, per-node trace buffer

---

Key Decisions

1. Zero OS dependencies. raft.h and raft.c depend only on stdint.h, string.h, and errno.h. No pthread.h, no atomics, no OS primitives. The library is a pure function of its raft_state_t input — all threading, locking, and scheduling are the caller's responsibility. In the controller adapter, the caller holds ha->lock around every raft_* call. In the deterministic simulator tests, no lock is needed (single-threaded). This makes the library embeddable in any threading model without modification.

2. Self-vote recorded immediately in raft_start_election. This prevents a concurrent node from granting its vote to two candidates (the root cause of observed split-brain). The self-vote is voted_for = self_idx at term increment time, before any RPC is sent.

3. Role enum is separate from ctrl_role_t. The Raft roles (FOLLOWER/CANDIDATE/LEADER) are a subset of XCE's controller states (INIT/STANDBY/ELECTING/PRIMARY_COMMITTING/PRIMARY). The adapter maps between them; the library owns only Raft roles.

4. Priority-based jitter is part of the library via raft_on_leader_lost. The jitter formula (priority_idx * election_timeout_ms) is in the library because it is a Raft-level correctness concern. The caller implements the actual sleep/timer.

5. Simulator routes callbacks synchronously. For test determinism, raft_sim delivers vote RPCs synchronously (no goroutines, no nanosleep). Timing tests verify term-based invariants, not wall-clock timing.