myxce/.kiro/specs/raft-library/design.md

# Design Document: raft-library

## Overview

This feature extracts the Raft-style election state machine from `ha.c` into a self-contained
C library (`src/lib/raft/`) that is decoupled from the iomgr/SSL/capnp transport layer. The
library exposes a pure state machine with a callback interface for all transport effects, enabling
deterministic unit testing without a live cluster.

**Purpose**: Provide a correct, independently verifiable Raft election implementation that eliminates
split-brain bugs traceable to entangled state transitions in `ha.c`.

**Users**: Controller developers will use this to wire the election protocol into the controller
via a thin adapter, and to verify protocol correctness via a deterministic N-node simulator.

**Impact**: `ha.c` becomes an adapter over `raft.h` rather than the owner of election state.
`ha_state_t.term`, `ha_state_t.voted_for_*`, and quorum logic move to `raft_state_t`.

### Goals
- Correct Raft invariants: one vote per term, quorum safety, no split-brain in 3-node deterministic tests
- Transport-free public API: `raft.h` has zero dependency on `endpoint_t`, `SSL_CTX`, `iomgr_t`, `cluster_t`
- Exhaustive protocol-level test cases covering all invariants specified in requirements
- Drop-in wiring: the controller adapter preserves existing XCE_TESTING hook seams

### Non-Goals
- Full Raft log replication (XCE is single-writer; only leader election is needed)
- WAL truncation on failover (remains in `ha.c`/`persist.c`)
- Config replication push (remains in `config_repl.c`)
- Disk persistence of term/voted_for (caller's responsibility via on_persist callback)
- Multi-Raft or multi-group election

---

## Boundary Commitments

### This Spec Owns
- `src/lib/raft/raft.h` — public API: `raft_state_t`, `raft_callbacks_t`, all function signatures
- `src/lib/raft/raft.c` — pure state machine implementation
- `src/lib/raft/raft_sim.c` — deterministic N-node simulator (test harness only)
- `src/lib/testsuite/test_raft.c` — protocol-level test suite (≥ 20 test cases)
- `src/lib/meson.build` additions for `raft.c` and `test_raft.c`
- The definition of `raft_callbacks_t` and the semantic contract for each callback

### Out of Boundary
- `ha.c` adapter wiring (separate task, touches ha.c but does not own raft.h)
- SSL/iomgr transport implementation of callbacks
- WAL truncation, config push, cluster_init_phase2
- Persistence to disk (ha-term.bin) — caller implements via on_persist callback
- Controller startup sequencing (`myapp_run`)

### Allowed Dependencies
- **`raft.h` and `raft.c`: `stdint.h`, `string.h`, `errno.h` only** — no pthread, no OS primitives
- The library has zero threading dependencies; concurrency is entirely the caller's responsibility
- No libxce dependencies from `raft.h` (headers that include `endpoint_t` or similar are forbidden)
- Test file may include `<unity/unity.h>` (project test framework) and `<pthread.h>` (for test harness only)
- `raft_sim.c` may include `<string.h>` and `<stdio.h>` for the simulator and dump log

### Revalidation Triggers
- Changes to `raft_callbacks_t` shape (adding/removing fields) require re-check of `ha.c` adapter
- Changes to quorum formula require re-running protocol-level tests
- Changes to role enum values require scanning `ha.c` for role-comparison code

---

## Architecture

### Existing Architecture Analysis

`ha.c` currently mixes three concerns:
1. **Raft state** — term, voted_for, role, quorum tracking (`ha_elect_ctx_t`)
2. **Transport** — `endpoint_connect()`, `iomgr_monitor()`, SSL setup, capnp encoding/decoding
3. **Post-election actions** — `cluster_init_phase2()`, `push_config_to_standbys()`, WAL truncation

The split-brain bugs arise from concern #1 being modified across threads without a consistent lock discipline: the transport layer (concern #2) and post-election hooks (concern #3) call into Raft state functions without a clear ownership model.

### Architecture Pattern

```mermaid
graph TB
    subgraph libxce_raft
        RaftState[raft_state_t]
        RaftAPI[raft.c Pure SM]
        RaftSim[raft_sim.c N-node sim]
    end

    subgraph ha_adapter
        HaAdapter[ha.c adapter]
    end

    subgraph transport
        Iomgr[iomgr endpoint_t SSL]
        Persist[ha_persist_term]
        DelayMgr[delaymgr timeout]
    end

    HaAdapter -->|calls| RaftAPI
    RaftAPI -->|callbacks| HaAdapter
    HaAdapter -->|drives| Iomgr
    HaAdapter -->|drives| Persist
    HaAdapter -->|drives| DelayMgr
    RaftSim -->|calls| RaftAPI
```

**Key decisions:**
- **No threads in the library**: `raft.c` is fully synchronous. The caller holds a lock before calling any `raft_*` function.
- **Callbacks fire synchronously**: The library calls back into the adapter within the same call frame. The adapter is responsible for not holding library locks during slow I/O.
- **Role enum is self-contained**: `raft_role_t` in `raft.h` is independent of `ctrl_role_t` in `ha.h`; the adapter maps between them.

### Technology Stack

| Layer | Choice | Role |
|-------|--------|------|
| Language | C gnu17 | Matches project standard |
| OS dependencies | **None** (`stdint.h`, `string.h`, `errno.h` only) | Library is OS-free; concurrency is entirely caller-owned |
| Threading | Caller-owned (pthread or none) | `raft_state_t` is not thread-safe by itself; caller holds whatever lock fits their threading model |
| Atomics | None in library | Caller uses atomics for ha_state_t integration if needed |
| Test framework | Unity (existing) | Consistent with controller testsuite |
| Build | Meson (existing) | `static_library('raft', 'raft.c')` added to `src/lib/meson.build` |

---

## File Structure Plan

```
src/lib/
├── raft/
│   ├── raft.h          # Public API: raft_state_t, raft_callbacks_t, all function signatures
│   ├── raft.c          # Pure state machine: term, voted_for, role, quorum evaluation
│   └── raft_sim.c      # N-node deterministic simulator (compiled only under XCE_TESTING)
├── testsuite/
│   └── test_raft.c     # Protocol-level test suite (≥ 20 cases)
└── meson.build         # Add raft.c to xcelib; add test_raft.c to xcelib_test
```

**Modified files:**
- `src/lib/meson.build` — add `raft/raft.c` to xcelib sources; add `testsuite/test_raft.c`
- `src/controller/ha.h` — add `#include "raft/raft.h"` and `raft_state_t raft` field to `ha_state_t` (adapter task)
- `src/controller/ha.c` — delegate election logic to `raft_*` API (adapter task, separate from library task)

---

## System Flows

### Election State Machine

```mermaid
stateDiagram-v2
    [*] --> FOLLOWER : raft_init
    FOLLOWER --> CANDIDATE : raft_start_election
    CANDIDATE --> LEADER : quorum reached
    CANDIDATE --> FOLLOWER : timeout or higher term
    LEADER --> FOLLOWER : higher term seen
    FOLLOWER --> FOLLOWER : raft_on_leader_lost clears voted_for
    FOLLOWER --> CANDIDATE : raft_on_election_timeout
```

### Vote Request Processing

```mermaid
sequenceDiagram
    participant Caller
    participant RaftSM as raft.c
    participant CB as callbacks

    Caller->>RaftSM: raft_on_vote_request(state, from, term)
    alt term < current_term
        RaftSM-->>Caller: RAFT_DENY stale term
    else term > current_term
        RaftSM->>RaftSM: update term, clear voted_for, role=FOLLOWER
    end
    alt already voted for different candidate same term
        RaftSM-->>Caller: RAFT_DENY already voted
    else on_vote_eligible returns 0
        RaftSM->>CB: on_vote_eligible(ctx, from, term)
        CB-->>RaftSM: 0 deny
        RaftSM-->>Caller: RAFT_DENY policy
    else all checks pass
        RaftSM->>CB: on_vote_eligible(ctx, from, term)
        CB-->>RaftSM: 1 grant
        RaftSM->>CB: on_persist(term, from)
        RaftSM->>RaftSM: voted_for = from
        RaftSM-->>Caller: RAFT_GRANT
    end
```

**on_vote_eligible contract** (implemented by ha.c adapter):
- Return 0 if local role is LEADER or CTRL_PRIMARY_COMMITTING
- Return 0 if STANDBY with fresh heartbeat (primary still alive)
- Return 0 if in-catch-up state
- Return 0 if candidate config_tag < local config_tag, or equal tag with lower LSN
- Return 1 otherwise (candidate is eligible)

### RDHUP Failover (Priority Jitter)

```mermaid
sequenceDiagram
    participant Caller
    participant RaftSM as raft.c
    participant CB as callbacks

    Caller->>RaftSM: raft_on_leader_lost(state, my_priority)
    RaftSM->>RaftSM: clear voted_for, role=FOLLOWER
    RaftSM->>CB: on_start_election_after_delay(priority * election_timeout_ms)
    note over Caller: delay elapses in caller's timer
    Caller->>RaftSM: raft_check_and_start_election(state)
    alt voted_for set to non-self (received vote during delay)
        RaftSM-->>Caller: RAFT_SKIP_ELECTION
    else
        RaftSM->>RaftSM: raft_start_election internally
        RaftSM->>CB: on_send_vote_request per peer
    end
```

---

## Components and Interfaces

### Component Summary

| Component | Layer | Intent | Req Coverage | Contracts |
|-----------|-------|--------|--------------|-----------|
| `raft_state_t` | Data | Raft node state | 1.1, 1.2 | State |
| `raft_callbacks_t` | Interface | Transport/effect boundary | 7.1, 7.2, 7.3 | Service |
| `raft.c` State Machine | Logic | Pure election logic | 1–6 | Service |
| `raft_sim.c` Simulator | Test | Deterministic N-node harness | 8.1, 8.2 | Service |
| `ha.c` Adapter | Integration | Wire raft.c to iomgr/SSL | 9.1, 9.2, 9.3 | Service |

---

### Core Data Structures

```c
/* raft_role_t: self-contained role enum (does not depend on ctrl_role_t) */
typedef enum {
    RAFT_FOLLOWER   = 0,
    RAFT_CANDIDATE  = 1,
    RAFT_LEADER     = 2,
} raft_role_t;

/* raft_state_t: pure Raft consensus state for one node.
 *
 * Contains only the Raft invariant state.  Application-specific fields
 * (config_tag, log_lsn, heartbeat timestamps, ctrl_role) stay in ha_state_t;
 * they are accessed by the caller via the on_vote_eligible callback. */
typedef struct {
    uint64_t      term;             /* current term (monotonically increasing) */
    raft_role_t   role;             /* current role */
    int           voted_for;        /* peer index voted for this term; -1=none; self_idx=self-vote */
    uint64_t      voted_for_term;   /* term in which voted_for was set */
    int           n_peers;          /* total cluster size (including self) */
    int           self_idx;         /* this node's index in [0, n_peers) */
    int           votes_granted;    /* votes collected in current election */
    int           election_decided; /* 1 once on_become_leader/follower fired */
    uint32_t      election_timeout_ms;
    int           my_priority;      /* priority index for jitter (0 = highest) */
} raft_state_t;

/* raft_callbacks_t: all side effects the library fires.
 *
 * The library owns Raft invariants (term ordering, one-vote-per-term, quorum).
 * All application-specific policy (freshness gate, role gate, catch-up state,
 * heartbeat liveness) is delegated to on_vote_eligible so the library is
 * fully independent of XCE domain logic.
 */
typedef struct {
    /* Send a RequestVote RPC to peer peer_idx for the given term.
     * The caller encodes config_tag, log_lsn, and any other fields in the wire RPC. */
    int  (*on_send_vote_request)(void *ctx, int peer_idx, uint64_t term);

    /* Application-specific vote eligibility gate.
     * Called after Raft invariants pass (correct term, voted_for unset).
     * Returns 1 to grant, 0 to deny.  Receives the candidate's peer_idx and
     * the incoming term so the caller can check freshness (config_tag, log_lsn),
     * role gate (LEADER/COMMITTING deny), catch-up status, heartbeat liveness.
     * The library grants the vote only if this callback returns 1. */
    int  (*on_vote_eligible)(void *ctx, int from_idx, uint64_t term);

    /* Node has won the election */
    void (*on_become_leader)(void *ctx, uint64_t term);
    /* Node lost election or saw higher term; leader_idx = -1 if unknown */
    void (*on_become_follower)(void *ctx, uint64_t term, int leader_idx);
    /* Persist term + voted_for before granting vote (durability guarantee) */
    int  (*on_persist)(void *ctx, uint64_t term, int voted_for);
    /* Schedule election timeout (fires raft_on_election_timeout after delay_ms) */
    void (*on_schedule_timeout)(void *ctx, uint32_t delay_ms);
    /* Cancel pending election timeout */
    void (*on_cancel_timeout)(void *ctx);
    /* Request delayed election start after RDHUP (priority-based jitter) */
    void (*on_start_election_after_delay)(void *ctx, uint32_t delay_ms);
    /* Trace callback for observability (optional, may be NULL) */
    void (*on_trace)(void *ctx, const char *event, const char *detail);
} raft_callbacks_t;
```

---

### Public API (`raft.h`)

```c
/*
 * raft_config_t: initialization context for a Raft node.
 *
 * Encapsulates all configuration so raft_init() has a single, extensible
 * parameter rather than a flat list.  Matches the existing XCE pattern
 * (ha_config_t, persist_config_t).  Zero-initialize and set only the
 * fields you need — the library fills in safe defaults for unset fields.
 *
 * Persistence model:
 *   initial_term / initial_voted_for are restored by the caller from disk
 *   before raft_init().  The library never reads storage; it only writes
 *   via on_persist callbacks.  Pass {0, -1} on first boot.
 */
typedef struct {
    int           n_peers;              /* total cluster size (including self) */
    int           self_idx;             /* this node's index in [0, n_peers) */
    uint32_t      election_timeout_ms;  /* base election timeout; 0 → default 2000ms */
    int           my_priority;          /* priority index for RDHUP jitter (0 = highest) */
    uint64_t      initial_term;         /* term restored from disk; 0 on first boot */
    int           initial_voted_for;    /* voted_for restored from disk; -1 if none */
    const raft_callbacks_t *cbs;        /* all callbacks; required callbacks must be non-NULL */
    void         *ctx;                  /* opaque caller context passed to every callback */
} raft_config_t;

/*
 * raft_init: initialize raft_state_t from a raft_config_t.
 * on_vote_eligible and on_trace may be NULL (optional policy / observability).
 * All other callbacks in cbs must be non-NULL.
 * Returns 0 on success; -EINVAL if a required callback is NULL or n_peers < 1.
 */
int raft_init(raft_state_t *s, const raft_config_t *cfg);

/*
 * raft_start_election: begin a new election.
 * Increments term, records self-vote (voted_for=self_idx), transitions to
 * CANDIDATE, fires on_send_vote_request for each peer, fires on_schedule_timeout.
 * Returns 0 on success; -EALREADY if already CANDIDATE/LEADER;
 * RAFT_ELECTED_IMMEDIATELY for single-node (n_peers==1).
 */
int raft_start_election(raft_state_t *s);

/*
 * raft_on_vote_request: process an incoming RequestVote RPC.
 * Applies Raft invariants (term comparison, one-vote-per-term).
 * If invariants pass, calls on_vote_eligible to allow application policy.
 * Fires on_persist before granting.
 * Returns RAFT_GRANT (1) or RAFT_DENY (0); sets *reply_term to current term.
 */
int raft_on_vote_request(raft_state_t *s, int from_idx, uint64_t term,
                          uint64_t *reply_term);

/*
 * raft_on_vote_reply: process a reply to our RequestVote.
 * Returns RAFT_BECAME_LEADER, RAFT_BECAME_FOLLOWER, or RAFT_PENDING.
 */
int raft_on_vote_reply(raft_state_t *s, int from_idx, int granted, uint64_t term);

/*
 * raft_on_election_timeout: called by the timer when no quorum was reached.
 * Transitions CANDIDATE → FOLLOWER; fires on_become_follower, on_cancel_timeout.
 */
void raft_on_election_timeout(raft_state_t *s);

/*
 * raft_on_leader_lost: primary control channel dropped (RDHUP).
 * Clears voted_for, transitions to FOLLOWER, fires on_start_election_after_delay.
 */
void raft_on_leader_lost(raft_state_t *s);

/*
 * raft_check_and_start_election: call after the delay from raft_on_leader_lost.
 * Starts election only if we have not already voted for a non-self candidate
 * in the new term (i.e., another node won during the jitter window).
 * Returns RAFT_SKIP_ELECTION or RAFT_ELECTION_STARTED.
 */
int raft_check_and_start_election(raft_state_t *s);

/* Return codes */
#define RAFT_OK                  0
#define RAFT_GRANT               1
#define RAFT_DENY                0
#define RAFT_BECAME_LEADER       1
#define RAFT_BECAME_FOLLOWER     2
#define RAFT_PENDING             3
#define RAFT_ELECTED_IMMEDIATELY 4
#define RAFT_SKIP_ELECTION       5
#define RAFT_ELECTION_STARTED    6
```

**Preconditions for all functions:** Caller must hold a mutex protecting `raft_state_t`.
**Postconditions:** Callbacks may fire synchronously within the call; caller must not hold transport locks during callback execution.

---

### Deterministic Simulator (`raft_sim.c`)

The simulator replaces real network I/O with synchronous in-process callback routing.

```c
/* raft_sim_node_t: one simulated cluster node */
typedef struct {
    raft_state_t   state;
    raft_callbacks_t cbs;
    void          *ctx;   /* points back to sim_cluster_t */
    int            idx;
    int            partition; /* 1 = isolated from other nodes */
} raft_sim_node_t;

/* raft_sim_cluster_t: N-node simulation cluster */
typedef struct {
    raft_sim_node_t nodes[RAFT_SIM_MAX_NODES];
    int             n;
    int             leader_count; /* incremented on each on_become_leader */
} raft_sim_cluster_t;

int  raft_sim_init(raft_sim_cluster_t *c, int n_nodes, uint32_t election_timeout_ms);
void raft_sim_start_all_elections(raft_sim_cluster_t *c); /* triggers all nodes simultaneously */
void raft_sim_deliver_pending(raft_sim_cluster_t *c);     /* routes buffered RPCs */
void raft_sim_fire_timeout(raft_sim_cluster_t *c, int node_idx);
void raft_sim_partition(raft_sim_cluster_t *c, int node_idx, int isolated);
```

---

## Protocol-Level Use Cases (Testing Strategy)

The following test cases verify Raft invariants at the protocol level, independent of transport.
Each test uses `raft_sim_cluster_t` and runs to completion synchronously.

### Group A: Basic Election Correctness (Req 1–4, 7, 8)

| TC | Scenario | Nodes | Expected |
|----|----------|-------|----------|
| A1 | Single-node cluster starts election | 1 | Immediately LEADER, on_become_leader fires once |
| A2 | 3-node simultaneous election, node 0 has priority | 3 | Exactly 1 LEADER; nodes 1,2 grant to node 0 and become FOLLOWER |
| A3 | 3-node election, node 0 wins, node 1 starts late election same term | 3 | Node 1's election denied (voted_for set); still 1 LEADER |
| A4 | 5-node election, quorum = 3 | 5 | Leader wins after 3 grants; 2 remaining nodes become FOLLOWER |
| A5 | Node receives vote request in LEADER role | 3 | Denies regardless of term; role stays LEADER |

### Group B: Voted_For Invariant (Req 3, one-vote-per-term)

| TC | Scenario | Expected |
|----|----------|----------|
| B1 | Node A and B both request vote from node C in same term | C grants only the first requester; second is denied |
| B2 | Node sees vote request with term > current_term | Updates term, clears voted_for, re-evaluates request |
| B3 | Node sees vote request with term < current_term | Denies immediately, returns current term |
| B4 | Node already voted for X in term T; receives request from Y in term T | Denies Y; voted_for stays X |
| B5 | Node granted vote for X in term T; receives request from X again in term T | Grants (idempotent re-grant to same candidate) |

### Group C: on_vote_eligible Policy Callback (Req 3.4, 3.5)

The library calls `on_vote_eligible` after Raft invariants pass. These tests use a
mock `on_vote_eligible` that returns a configurable value:

| TC | Scenario | on_vote_eligible returns | Expected |
|----|----------|--------------------------|----------|
| C1 | All Raft invariants pass; on_vote_eligible returns 0 | 0 | Denied (policy) |
| C2 | All Raft invariants pass; on_vote_eligible returns 1 | 1 | Granted |
| C3 | on_vote_eligible is NULL at init | — | All eligible candidates granted (no policy gate) |
| C4 | Raft invariant fails (stale term); on_vote_eligible NOT called | N/A | Denied before callback fires |
| C5 | Voted-for already set; on_vote_eligible NOT called | N/A | Denied before callback fires |

*Note*: The freshness gate (config_tag, log_lsn), role gate (LEADER/COMMITTING deny),
and heartbeat-freshness gate are ha.c adapter responsibilities implemented inside
`on_vote_eligible` — not tested in the library test suite.

### Group H: Persistence and Restart (Req 1.2, on_persist)

| TC | Scenario | Expected |
|----|----------|----------|
| H1 | raft_init with cfg.initial_term=5, cfg.initial_voted_for=1 | state.term=5, state.voted_for=1 immediately |
| H2 | raft_init with cfg.initial_term=0, cfg.initial_voted_for=-1 (first boot) | state.term=0, state.voted_for=-1 |
| H3 | Vote granted; on_persist fires before voted_for is recorded | on_persist called with (term, from_idx) before grant returns |
| H4 | on_persist returns non-zero (disk failure) | Vote denied; voted_for unchanged |
| H5 | raft_start_election: on_persist fires for self-vote | on_persist called with (new_term, self_idx) |

### Group D: Quorum and Result (Req 4)

| TC | Scenario | Expected |
|----|----------|----------|
| D1 | 3-node: 1 grant received (self + 1 = 2 = quorum) | LEADER immediately |
| D2 | 3-node: both peers deny | election_decided=1, FOLLOWER, on_become_follower fires |
| D3 | on_become_leader fires: second vote reply arrives | No-op (election_decided guard) |
| D4 | Reply with term > current_term | Update term, FOLLOWER, on_become_follower fires |

### Group E: Election Timeout (Req 6)

| TC | Scenario | Expected |
|----|----------|----------|
| E1 | Timeout fires before quorum | on_become_follower; on_cancel_timeout |
| E2 | Timeout fires after quorum | No-op (election already decided) |
| E3 | New election after timeout | term increments; on_schedule_timeout fires again |

### Group F: RDHUP Failover (Req 5)

| TC | Scenario | Expected |
|----|----------|----------|
| F1 | 3-node: node 0 killed; nodes 1,2 call raft_on_leader_lost | Node 1 (priority=1) waits 1×timeout; node 2 (priority=2) waits 2×timeout |
| F2 | During node 1's delay, node 1 receives vote request from another election starter | voted_for set; raft_check_and_start_election returns RAFT_SKIP_ELECTION |
| F3 | During node 1's delay, no vote request arrives | raft_check_and_start_election starts election; 1 LEADER |
| F4 | Simultaneous RDHUP (all remaining nodes fire at same time, no delay difference) | Only one LEADER (self-vote prevents double grant) |

### Group G: Split-Brain Prevention (Req 4, overall correctness)

| TC | Scenario | Expected |
|----|----------|----------|
| G1 | 3-node simultaneous elections all terms (100 random election rounds via simulator) | Never more than 1 LEADER per term across all rounds |
| G2 | Network partition: nodes 0+1 isolated from node 2 | Partition of 2 can elect; partition of 1 cannot |

---

## Requirements Traceability

| Req | Summary | Component | Interface/Contract |
|-----|---------|-----------|-------------------|
| 1.1 | State fields | `raft_state_t` | Data struct |
| 1.2 | Init to FOLLOWER | `raft_init` | Function |
| 1.3 | No transport types in header | `raft.h` boundary | Boundary commit |
| 2.1 | Election: incr term, self-vote, send RPCs | `raft_start_election` | Function |
| 2.2 | Single-node shortcircuit | `raft_start_election` | `RAFT_ELECTED_IMMEDIATELY` |
| 2.3 | Error if already CANDIDATE/LEADER | `raft_start_election` | `-EALREADY` |
| 3.1 | Higher term → update, re-evaluate | `raft_on_vote_request` | Function |
| 3.2 | Lower term → deny | `raft_on_vote_request` | Return value |
| 3.3 | Same term, different voted_for → deny | `raft_on_vote_request` | voted_for check |
| 3.4 | Freshness gate → grant + persist | `raft_on_vote_request` | `on_persist` callback |
| 3.5 (5) | LEADER/COMMITTING → deny | `raft_on_vote_request` | Role check |
| 4.1 | Quorum → LEADER | `raft_on_vote_reply` | `RAFT_BECAME_LEADER` |
| 4.2 | Higher term in reply → FOLLOWER | `raft_on_vote_reply` | `RAFT_BECAME_FOLLOWER` |
| 4.3 | Post-decision no-op | `raft_on_vote_reply` | `election_decided` guard |
| 4.4 | All replied, no quorum → FOLLOWER | `raft_on_vote_reply` | vote counting |
| 5.1 | Leader lost → clear voted_for + delay callback | `raft_on_leader_lost` | `on_start_election_after_delay` |
| 5.2 | Already voted during delay → skip | `raft_check_and_start_election` | `RAFT_SKIP_ELECTION` |
| 5.3 | Not voted → start election | `raft_check_and_start_election` | `RAFT_ELECTION_STARTED` |
| 6.1 | Start election → schedule timeout | `raft_start_election` | `on_schedule_timeout` |
| 6.2 | Timeout fires, undecided → follower | `raft_on_election_timeout` | `on_become_follower` |
| 6.3 | Timeout fires, decided → no-op | `raft_on_election_timeout` | `election_decided` guard |
| 7.1 | raft_callbacks_t struct definition | `raft.h` | Data struct |
| 7.2 | void* ctx threading | All callbacks | Function signatures |
| 7.3 | NULL callback → error | `raft_init` | `-EINVAL` |
| 8.1 | Deterministic simulator | `raft_sim.c` | Simulator API |
| 8.2 | 4 invariant test cases | `test_raft.c` | Groups A–G |
| 8.3 | ASAN clean | Build config | Meson test suite |
| 9.1 | Adapter implements callbacks | `ha.c` adapter (separate task) | raft_callbacks_t impl |
| 9.2 | ha.c delegates state to library | `ha.c` adapter (separate task) | ha_state_t.raft field |
| 9.3 | XCE_TESTING hooks preserved | `ha.c` adapter (separate task) | Existing seams |

---

## Error Handling

| Error | Source | Response |
|-------|--------|----------|
| NULL callback at init | `raft_init` | Return `-EINVAL` before any state is set |
| `raft_start_election` called while CANDIDATE | State check | Return `-EALREADY` |
| `on_persist` callback returns error | `raft_on_vote_request` | Do not grant vote; return RAFT_DENY |
| `on_send_vote_request` fails for one peer | Election start | Count as no-reply for that peer; election proceeds with remaining peers |

---

## Observability and Debug Tracing

Protocol correctness requires that every state transition is observable without a live cluster.
The library provides two tracing mechanisms:

### 1. Trace Callback (production + test)

Add an optional `on_trace` field to `raft_callbacks_t`:

```c
/* on_trace: fired on every state transition or significant decision.
 * event: short string literal, e.g. "vote_granted", "became_leader", "deny_stale_term"
 * detail: formatted human-readable context string (caller-allocated)
 * May be NULL to suppress tracing. */
void (*on_trace)(void *ctx, const char *event, const char *detail);
```

Every call to a public `raft_*` function fires `on_trace` at entry and at each decision point:

| Trace event | When fired |
|-------------|-----------|
| `election_start` | `raft_start_election` begins |
| `self_vote` | Self-vote recorded |
| `vote_request_in` | `raft_on_vote_request` called |
| `vote_granted` | Vote granted to candidate |
| `vote_denied_stale_term` | Denied: term < current |
| `vote_denied_already_voted` | Denied: voted_for already set |
| `vote_denied_role` | Denied: LEADER or COMMITTING |
| `vote_denied_freshness` | Denied: candidate not fresh |
| `vote_reply_in` | `raft_on_vote_reply` called |
| `quorum_reached` | Votes_granted >= quorum |
| `became_leader` | on_become_leader fired |
| `became_follower` | on_become_follower fired |
| `election_timeout` | `raft_on_election_timeout` called |
| `leader_lost` | `raft_on_leader_lost` called |
| `jitter_skip` | `raft_check_and_start_election` → SKIP |
| `term_update` | term incremented or bumped by peer |

In production (`ha.c` adapter), `on_trace` routes to `INFO(...)` logging.
In tests, `on_trace` appends to a per-node trace buffer; test assertions can inspect the
event sequence to verify protocol correctness beyond role/term checks.

### 2. State Snapshot (`raft_snapshot`)

```c
/* raft_snapshot_t: point-in-time read of all Raft state for logging/debugging */
typedef struct {
    uint64_t     term;
    raft_role_t  role;
    int          voted_for;
    uint64_t     voted_for_term;
    int          votes_granted;
    int          election_decided;
} raft_snapshot_t;

/* Fills snap with current state. Thread-safe (caller must hold lock). */
void raft_snapshot(const raft_state_t *s, raft_snapshot_t *snap);
```

The `raft_snapshot` function is used by:
- The simulator to print cluster state after each test step
- The adapter to log state on RDHUP for post-mortem debugging
- The ASAN test runner to dump state on test failure

### 3. Simulator Trace Output

`raft_sim_cluster_t` records a timestamped event log for each RPC and state transition.
After each test, `raft_sim_dump_log(cluster, FILE*)` prints the protocol trace in a
human-readable format, e.g.:

```
[step 0] node0: election_start term=2
[step 1] node0→node1: vote_request term=2 tag=0 lsn=0
[step 1] node0→node2: vote_request term=2 tag=0 lsn=0
[step 2] node1: vote_granted to node0 term=2
[step 2] node2: vote_granted to node0 term=2
[step 3] node0: quorum_reached votes=3 → became_leader term=2
```

This trace is printed to stderr on test failure and can be enabled unconditionally
with `RAFT_SIM_VERBOSE=1`.

### File structure additions

```
src/lib/raft/
├── raft.h          # + on_trace, raft_snapshot_t, raft_snapshot()
├── raft.c          # + trace emission at every decision point
└── raft_sim.c      # + raft_sim_dump_log, per-node trace buffer
```

---

## Key Decisions

1. **Zero OS dependencies.** `raft.h` and `raft.c` depend only on `stdint.h`, `string.h`, and `errno.h`. No `pthread.h`, no atomics, no OS primitives. The library is a pure function of its `raft_state_t` input — all threading, locking, and scheduling are the caller's responsibility. In the controller adapter, the caller holds `ha->lock` around every `raft_*` call. In the deterministic simulator tests, no lock is needed (single-threaded). This makes the library embeddable in any threading model without modification.

2. **Self-vote recorded immediately in `raft_start_election`.** This prevents a concurrent node from granting its vote to two candidates (the root cause of observed split-brain). The self-vote is `voted_for = self_idx` at term increment time, before any RPC is sent.

3. **Role enum is separate from `ctrl_role_t`.** The Raft roles (FOLLOWER/CANDIDATE/LEADER) are a subset of XCE's controller states (INIT/STANDBY/ELECTING/PRIMARY_COMMITTING/PRIMARY). The adapter maps between them; the library owns only Raft roles.

4. **Priority-based jitter is part of the library via `raft_on_leader_lost`.** The jitter formula (`priority_idx * election_timeout_ms`) is in the library because it is a Raft-level correctness concern. The caller implements the actual sleep/timer.

5. **Simulator routes callbacks synchronously.** For test determinism, `raft_sim` delivers vote RPCs synchronously (no goroutines, no nanosleep). Timing tests verify term-based invariants, not wall-clock timing.