# Design: HA Config Bootstrap & Raft-Style Quorum Election

## Overview

This feature eliminates three critical safety gaps in the XCE controller's LSF-style priority-based election: unconfigured nodes can win elections and serve stale cluster state; concurrent-startup nodes falsely self-elect because the peer listener is not yet open; and a promoted standby with missing local config has no bootstrap path. The fix replaces priority election with a Raft-style quorum protocol that encodes config freshness in votes, moves the listener before election, and adds majority-commit durability for both WAL writes and config upgrades.

**Purpose**: Delivers full CP high-availability to cluster operators — zero data loss on failover, no split-brain on concurrent startup, and no silent config loss on primary crash. The design chooses Consistency over Availability (CAP-CP): a minority partition cannot elect a primary; the cluster waits until majority is restored.

**Impact**: Replaces `am_i_primary()` and `ha_check_heartbeat_nocluster()` with Raft quorum election and `EPOLLRDHUP`-based failover; extends the controller lifecycle from a 2-state (`CTRL_PRIMARY`/`CTRL_STANDBY`) to a 4-state FSM adding `CTRL_INIT` and `CTRL_ELECTING`; adds majority-commit gating to `persist_wal_sync_append()` and `push_config_to_standbys()`; establishes a persistent TLS control channel between each standby and the primary; adds transparent retry to libxce.

### Goals

- Raft-style quorum election with config-freshness tiebreaking (CP semantics, no minority-partition primaries)
- Zero data loss on failover via majority-commit WAL and config replication
- Sub-second failover detection via `EPOLLRDHUP` + heartbeat timeout vtask
- Transparent recovery for unconfigured nodes joining an existing cluster (deferred config fetch)
- Transparent client retry in libxce — worker/scheduler/apiserver code unchanged

### Non-Goals

- Raft snapshot protocol (existing dual-zarray COW is orthogonal and unchanged)
- Joint consensus for dynamic controller membership changes (`--peers` list is static at startup)
- mTLS between controllers (server-side TLS sufficient for trusted-LAN model)
- Config file format or storage layout changes
- Worker/scheduler daemon protocol changes

---

## Boundary Commitments

### This Spec Owns

- Controller election protocol: full replacement of `am_i_primary()` with `am_i_primary_quorum()` (Note: `am_i_primary_quorum()`'s blocking pthread-per-peer implementation was itself later removed — production election now runs through async `ha_start_election_async()` delegating to the `src/lib/raft/` library's `raft_start_election()`; superseded by raft-library and election-iomgr-refactor, see those specs' design.md)
- Controller FSM: 4-state lifecycle (`CTRL_INIT`, `CTRL_ELECTING`, `CTRL_PRIMARY`, `CTRL_STANDBY`) and all state transitions
- Pre-election startup sequence: listener binding, iomgr/vtaskmgr init before election
- Config bootstrap: `bootstrap_config_if_needed()`, `xce_probe_config_tag()`, deferred fetch after election
- Startup validation: `--vcpus N` CLI, even-N peer count rejection, TLS cert pre-flight, ASAN leak fix
- Majority-commit WAL: `repl_broadcast()`, `repl_wait_majority_ack()`, `committed_lsn`, `persist_truncate_wal_above()`
- Majority-commit config push: `OP_upgradeConfigAck`, `committed_config_tag`, heartbeat piggybacking
- Persistent control channel: `ha_find_and_connect_to_primary()`, `EPOLLRDHUP` failover, WAL streaming over channel, three-tier primary discovery
- Election state durability: `ha-term.bin`, `ha_persist_term()`, `voted_for_term/name` across restarts
- libxce client retry: `xce_retry_config_t`, retry wrapper, `XCE_ERR_STANDBY` redirect fields in wire protocol
- Wire protocol extensions: `OP_requestVote`, `OP_probeConfigTag`, `OP_walAck`, `OP_upgradeConfigAck`, `OP_snapshotTransfer`, `OP_walCatchupEntry`
- Structured log observability for all new code paths (election, bootstrap, WAL commit, config push, FSM transitions)

### Out of Boundary

- `apply_wal_entry()` — must be extracted from the startup WAL replay path as a prerequisite to Change F; not authored in this spec
- `persist_checkpoint()` trigger policy — existing on-demand call in `repl_needs_snapshot()` path; unchanged
- Worker/scheduler daemon code — libxce retry is transparent; no caller code changes
- apiserver Go tier — no changes
- `scheduler-controller-msg-gap` spec and `worker-join-refactor` spec — orthogonal; no shared files
- Dynamic controller membership changes (joint consensus)
- mTLS between controllers

### Allowed Dependencies

- `iomgr` (epoll/`EPOLLIN`/`EPOLLRDHUP`): P0 — control channel and dispatch loop
- `vtaskmgr`: P0 — `vtaskmgr_init(int nvcpus)` confirmed to take vcpus directly, no cluster_config dependency
- `endpoint_connect()` / `connect_ssl_raw()`: P0 — SSL connection setup; TCP connect phase is blocking; vote request connections require per-peer pthreads (Note: superseded by election-iomgr-refactor — vote requests now use async iomgr endpoints, not per-peer pthreads; see that spec's design.md)
- `repl_add/remove/start_catchup/complete_catchup/needs_snapshot/on_ack/trim_lsn`: P0 — existing repl slot management reused unchanged
- `do_one_config_fetch()`: P1 — wired on branch (`2aae45f`); deferred fetch depends on it
- `xce_encode()` / `xce_decode()` / `xce_request()`: P0 — wire protocol encoding
- `ha_do_promote()` / `ha_do_demote()`: P0 — role transitions, reused
- `src/schema/controller.capnp`: P0 — new opcodes appended after existing @31
- `xcelog.h` INFO/WARN/ERROR macros: P2 — observability

### Revalidation Triggers

- Changes to `ha_state_t` field layout require re-checking all ha.c callers
- Changes to `RequestReply` or `CtrlHeartbeat` capnp structs require checking all decode sites
- Moving `cluster_open_listener()` before election changes startup order; any code assuming phase2 runs before network is up must be re-verified
- `committed_lsn` cross-cuts `persist_t`, `repl.c`, and `ha_state_t`; changes to any require re-checking the others
- `ha-term.bin` format change requires migration or version field

---

## Architecture

### Existing Architecture Analysis

The current controller has a 2-state election: `am_i_primary()` spawns per-peer pthreads that probe higher-priority peers and self-elect on timeout. `cluster_init_phase2()` contains `bind()+listen()`, `iomgr_init()`, and `vtaskmgr_init()` — all run after election, creating a 2–3 second window where the port is unreachable to election probes.

Confirmed gaps from codebase inspection (see `research.md` §1 for full inventory):
- `ctrl_role_t` has only `CTRL_PRIMARY=0`, `CTRL_STANDBY=1`
- `ha_state_t` lacks: `voted_for_term`, `voted_for_name`, `primary_ctrl_fd`, `defer_config_fetch`, `elect_ctx`, `last_known_committed_lsn`, `heartbeat_seen`, `last_known_committed_config_tag`
- `repl_broadcast()` is a stub returning 0 without sending; `repl_wait_majority_ack()` does not exist
- `persist_t` lacks `committed_lsn`; `persist_truncate_wal_above()` does not exist
- `XCE_ERR_STANDBY` reply has no `redirect_host/port` fields — must be added to capnp schema
- `ha-term.bin` does not exist; `ha_init()` always starts from term=0
- `ha_check_priority_stepdown()` is permanently unreachable after first quorum election (term always ≥ 1)
- `ha_check_heartbeat_nocluster()` uses 10s timestamp polling — replaced by dual-trigger failover

### Architecture Pattern & FSM

**Selected pattern**: 4-state FSM with Raft-style majority-quorum consensus (CP). Each startup step gates the next.

```mermaid
stateDiagram-v2
    [*] --> CTRL_INIT : startup complete listener open
    CTRL_INIT --> CTRL_STANDBY : heartbeat received from existing primary
    CTRL_INIT --> CTRL_ELECTING : election timeout fires
    CTRL_ELECTING --> CTRL_PRIMARY : quorum won and no fresher peer
    CTRL_ELECTING --> CTRL_STANDBY : quorum lost or fresher peer yields
    CTRL_STANDBY --> CTRL_ELECTING : EPOLLRDHUP or heartbeat timeout vtask
    CTRL_PRIMARY --> CTRL_STANDBY : higher-term reply received
```

```mermaid
graph TB
    subgraph Startup
        V[validate_peer_count + validate_certs] --> P1[cluster_init_phase1]
        P1 --> Net[iomgr_init + cluster_open_listener + vtaskmgr_init]
        Net --> Boot[bootstrap_config_if_needed]
        Boot --> Wait[ha_wait_for_heartbeat_or_timeout]
    end
    subgraph Election
        Wait --> Quorum[am_i_primary_quorum]
        Quorum --> Vote[OP_requestVote per peer thread]
        Vote --> Eval[ha_elect_evaluate_and_signal]
    end
    subgraph Replication
        WAL[persist_wal_sync_append] --> Bcast[repl_broadcast]
        Bcast --> MAck[repl_wait_majority_ack]
        MAck --> Commit[advance committed_lsn]
    end
    subgraph ControlChannel
        SB[CTRL_STANDBY] --> Conn[ha_find_and_connect_to_primary]
        Conn --> Epoll[EPOLLIN + EPOLLRDHUP]
        Epoll --> Sync[state sync WAL streaming]
    end
```

**Note (election mechanism superseded)**: The diagrams above show this spec's original blocking pthread-per-peer `am_i_primary_quorum()` election path. That implementation was removed by election-iomgr-refactor (task 5.3) and replaced by async `ha_start_election_async()` delegating to the `src/lib/raft/` library's `raft_start_election()`; `handle_request_vote()` is now test-only, production vote dispatch is `cluster.c` → `raft_on_vote_request()`. The 4-state FSM, majority-commit, and control-channel shapes shown remain accurate. See election-iomgr-refactor and raft-library design.md for the current mechanism.

### Technology Stack

| Layer | Choice | Role | Notes |
|-------|--------|------|-------|
| Runtime | C gnu17 | All new code | No C++ |
| Wire format | Cap'n Proto | 6 new opcodes in `controller.capnp` | Appended after existing @31 |
| SSL | OpenSSL (existing) | Vote request and probe connections | Reuses `connect_ssl_raw()` |
| Consensus | Raft subset (custom) | Leader election + majority-commit | No external library; bounded scope justified by static membership and no snapshot protocol need (Note: a dedicated `src/lib/raft/` in-repo library was later extracted from this inline logic; superseded by raft-library, see that spec's design.md) |
| IPC | iomgr epoll | `EPOLLIN` + `EPOLLRDHUP` on control channel | `EPOLLRDHUP` for sub-second failover |
| Persistence | Flat binary (`ha-term.bin`) | Term + voted_for across restarts | Atomic write+rename (existing pattern) |
| Build | Meson + Nix | capnp schema regeneration | `src/schema/controller.capnp` |

---

## File Structure Plan

### Directory Structure

```
src/
├── schema/
│   └── controller.capnp               # Modified: 6 new opcodes + message structs
├── controller/
│   ├── ha.h                           # Modified: CTRL_INIT/CTRL_ELECTING enum values;
│   │                                  #   ha_vote_entry_t, ha_elect_ctx_t types;
│   │                                  #   new ha_state_t fields; ha_persist_term() decl
│   ├── ha.c                           # Modified: replace am_i_primary() -> am_i_primary_quorum();
│   │                                  #   add handle_request_vote(), handle_request_vote_reply(),
│   │                                  #   ha_wait_for_heartbeat_or_timeout(), ha_elect_evaluate_and_signal(),
│   │                                  #   ha_find_and_connect_to_primary(), ha_persist_term();
│   │                                  #   retire ha_check_priority_stepdown(), ha_check_heartbeat_nocluster();
│   │                                  #   SSL_CTX_free ASAN fix in ha_fini()
│   ├── persist.h                      # Modified: add committed_lsn to persist_t
│   ├── persist.c                      # Modified: persist_truncate_wal_above() (new);
│   │                                  #   upgrade persist_wal_sync_append() to repl_wait_majority_ack()
│   ├── repl.h                         # Modified: repl_wait_majority_ack() decl;
│   │                                  #   pending_ack_count in standby_conn_t
│   ├── repl.c                         # Modified: implement repl_broadcast() (was stub);
│   │                                  #   repl_wait_majority_ack() (new); backpressure; catch-up exclusion
│   ├── config_repl.h                  # Modified: bootstrap_config_if_needed() and xce_probe_config_tag() decls
│   ├── config_repl.c                  # Modified: bootstrap_config_if_needed() (new);
│   │                                  #   xce_probe_config_tag() (new);
│   │                                  #   upgrade push_config_to_standbys() to majority-commit
│   ├── cluster.h                      # Modified: remove vcpus from cluster_config_t
│   ├── cluster.c                      # Modified: extract cluster_open_listener() from phase2;
│   │                                  #   extended handle_ctrlhello(); OP_snapshotTransfer and
│   │                                  #   OP_walCatchupEntry dispatch handlers
│   ├── myapp.c                        # Modified: new startup order (listener before election);
│   │                                  #   bootstrap_config_if_needed() call; deferred fetch integration
│   ├── main.c                         # Modified: --vcpus N CLI arg; even-N peer count validation;
│   │                                  #   cert pre-flight access() check
│   └── testsuite/
│       ├── test_ha_election.c         # New: quorum election, vote protocol, FSM transitions
│       ├── test_ha_bootstrap.c        # New: bootstrap_config_if_needed, probe RPC, deferred fetch
│       ├── test_ha_durability.c       # New: ha-term.bin persist/restore, CRC corruption
│       ├── test_persist_truncate.c    # New: persist_truncate_wal_above, boundary segment, idempotency
│       ├── test_repl_majority.c       # New: repl_broadcast, repl_wait_majority_ack, backpressure
│       └── test_ha_startup.c          # New: startup validation (vcpus, peer count, cert pre-flight)
└── lib/
    ├── request.h                      # Modified: xce_retry_config_t; xce_request_with_retry() decl
    ├── request.c                      # Modified: retry wrapper; redirect handling; backoff logic
    └── testsuite/
        └── test_libxce_retry.c        # New: retry logic, backoff, redirect, non-retriable errors
```

### Modified Files Summary

| File | Change Type | Requirements |
|------|-------------|--------------|
| `src/schema/controller.capnp` | Extend: 6 opcodes + structs | 2, 3, 5, 6, 7, 8 |
| `src/controller/ha.h` | Extend: enum + types + fields | 1, 2, 3, 5, 6, 9 |
| `src/controller/ha.c` | Major: replace election core; add control channel | 1, 2, 3, 6, 9 |
| `src/controller/persist.h` | Extend: committed_lsn | 5 |
| `src/controller/persist.c` | New function + upgrade sync path | 5 |
| `src/controller/repl.h` | Extend: standby_conn_t field + decl | 5, 8 |
| `src/controller/repl.c` | Implement stubs + add majority logic | 5, 6, 8 |
| `src/controller/config_repl.h` | Extend: two new function decls | 3, 8 |
| `src/controller/config_repl.c` | New functions + upgrade push | 3, 8 |
| `src/controller/cluster.h` | Remove vcpus field | 4 |
| `src/controller/cluster.c` | Extract listener; extend ctrlhello handler | 1, 6 |
| `src/controller/myapp.c` | New startup sequence | 1, 3 |
| `src/controller/main.c` | CLI validation + pre-flight | 4 |
| `src/lib/request.h` | New type + decl | 7 |
| `src/lib/request.c` | Retry wrapper | 7 |

---

## System Flows

### Flow 1: Controller Startup & Election

```mermaid
sequenceDiagram
    participant Main as main.c
    participant MyApp as myapp.c
    participant HA as ha.c
    participant Cluster as cluster.c
    participant Config as config_repl.c
    Main->>Main: validate_peer_count odd-N check
    Main->>Main: validate_certs access()
    Main->>MyApp: myapp_run()
    MyApp->>Cluster: cluster_init_phase1 reads ha-term.bin via ha_init
    MyApp->>Cluster: iomgr_init + cluster_open_listener + vtaskmgr_init
    Note over Cluster: listener open BEFORE election
    MyApp->>Config: bootstrap_config_if_needed
    Note over Config: tag=0 and peers have config? set defer_config_fetch and return
    MyApp->>HA: ha_wait_for_heartbeat_or_timeout
    alt heartbeat received in window
        HA->>MyApp: set CTRL_STANDBY skip election
    else election timeout fires
        HA->>HA: am_i_primary_quorum send OP_requestVote per peer
        HA->>MyApp: CTRL_PRIMARY or CTRL_STANDBY
    end
    alt defer_config_fetch and not primary
        MyApp->>Config: do_one_config_fetch from proven winner
    end
    MyApp->>Cluster: cluster_init_phase2_rest kvmgr jobmgr apiserver etc.
```

### Flow 2: Quorum Election Vote Protocol

```mermaid
sequenceDiagram
    participant Cand as Candidate ha.c
    participant PA as Peer A ha.c
    participant PB as Peer B ha.c
    Cand->>Cand: increment term persist ha-term.bin
    Cand->>PA: OP_requestVote term config_tag last_lsn via per-peer thread
    Cand->>PB: OP_requestVote term config_tag last_lsn via per-peer thread
    PA->>PA: handle_request_vote check voted_for and log_ok
    PA->>Cand: OP_requestVoteReply granted=true peer_tag peer_lsn
    PB->>PB: handle_request_vote check voted_for and log_ok
    PB->>Cand: OP_requestVoteReply granted=true peer_tag peer_lsn
    Cand->>Cand: ha_elect_evaluate_and_signal
    Note over Cand: votes_granted ge quorum AND no fresher peer? CTRL_PRIMARY
```

Key decisions: fresher-peer yield gate fires when any *responding* peer has higher `config_tag` or equal tag with higher LSN; non-responding peers never block election. Split-vote triggers one retry with incremented term and randomized jitter delay.

### Flow 3: WAL Majority-Commit

```mermaid
sequenceDiagram
    participant Caller as Job submit caller
    participant Persist as persist.c
    participant Repl as repl.c
    participant SA as Standby A
    participant SB as Standby B
    Caller->>Persist: persist_wal_sync_append WAL_JOB_SUBMIT
    Persist->>Persist: write + fsync local advance applied_lsn
    Persist->>Repl: repl_broadcast OP_walAck payload over persistent channel
    Repl->>SA: WAL entry on control channel fd
    Repl->>SB: WAL entry on control channel fd
    SA->>Repl: OP_walAck lsn=N
    Repl->>Persist: repl_wait_majority_ack quorum reached
    Persist->>Persist: advance committed_lsn to N
    Persist->>Caller: return 0 success
    SB->>Repl: OP_walAck lsn=N arrives later counted but not blocking
```

Entry types `WAL_HOST_STATE` and `WAL_CHECKPOINT_REF` bypass this flow via `persist_wal_async()` (best-effort, unchanged).

### Flow 4: Standby Failover Detection and Re-election

```mermaid
sequenceDiagram
    participant IOM as iomgr epoll
    participant HA as ha.c standby
    participant Peers as Peer controllers
    IOM->>HA: EPOLLRDHUP on primary_ctrl_fd
    HA->>HA: close primary_ctrl_fd set CTRL_ELECTING
    HA->>HA: persist_truncate_wal_above last_known_committed_lsn
    HA->>Peers: am_i_primary_quorum new term OP_requestVote
    alt quorum won
        HA->>HA: CTRL_PRIMARY accept inbound control channels
    else quorum lost
        HA->>Peers: ha_find_and_connect_to_primary three-tier discovery
        HA->>HA: CTRL_STANDBY
    end
```

Fallback trigger: heartbeat timeout vtask fires after `election_timeout_ms + rand_jitter` of silence on an open TCP channel (covers network partitions that keep TCP alive but suppress heartbeats).

---

## Requirements Traceability

| Requirement | Summary | Components | Key Interface | Flow |
|-------------|---------|------------|---------------|------|
| 1.1–1.4 | Listener + iomgr/vtaskmgr before election; dispatch gate for init/electing states | `cluster.c`, `ha.c`, `myapp.c` | `cluster_open_listener()`, vtaskmgr dispatch gate | Flow 1 |
| 2.1–2.12 | Quorum election: voted_for enforcement, freshness gate, split-vote retry, single-node shortcircuit | `ha.c` | `am_i_primary_quorum()`, `handle_request_vote()` (Note: both superseded by `ha_start_election_async()`/raft library, `handle_request_vote()` now test-only — see election-iomgr-refactor), `OP_requestVote` | Flow 2 |
| 3.1–3.5 | Pre-election config probe, deferred fetch, retry backoff | `config_repl.c`, `myapp.c` | `bootstrap_config_if_needed()`, `xce_probe_config_tag()`, `OP_probeConfigTag` | Flow 1 |
| 4.1–4.4 | --vcpus CLI, even-N rejection, cert pre-flight, ASAN fix | `main.c`, `ha.c` | CLI argp, `access()`, `SSL_CTX_free` | Flow 1 |
| 5.1–5.8 | Majority-commit WAL: committed_lsn, broadcast, ack-wait, backpressure, truncation | `persist.c`, `repl.c`, `ha.c` | `repl_wait_majority_ack()`, `persist_truncate_wal_above()`, `OP_walAck` | Flow 3 |
| 6.1–6.8 | Persistent control channel: EPOLLRDHUP failover, state sync, catch-up, three-tier discovery | `ha.c`, `cluster.c`, `repl.c` | `ha_find_and_connect_to_primary()`, extended `OP_ctrlHello`, `OP_snapshotTransfer`, `OP_walCatchupEntry` | Flow 4 |
| 7.1–7.6 | Client retry: backoff, redirect, non-retriable passthrough | `request.c` (libxce) | `xce_request_with_retry()` (Note: removed, superseded by `client_failover.c` StatusClassifier/EndpointRotator — see libxce-failover), `xce_retry_config_t`, redirect fields in `OP_requestVoteReply` | — |
| 8.1–8.5 | Majority-commit config push, committed_config_tag, heartbeat piggybacking | `config_repl.c`, `repl.c` | `push_config_to_standbys()`, `OP_upgradeConfigAck` | Flow 3 analog |
| 9.1–9.3 | Term + voted_for durability across restarts via ha-term.bin | `ha.c` | `ha_persist_term()`, `ha_init()` read path | — |
| 10.1–10.5 | Structured logs for election outcome, bootstrap, WAL commit, config push, FSM transitions | All new code paths | `xcelog.h` INFO/WARN macros | All flows |

---

## Components and Interfaces

### Summary Table

| Component | File(s) | Intent | Req Coverage | Key Dependencies |
|-----------|---------|--------|-------------|-----------------|
| Controller FSM | `ha.h`, `ha.c` | 4-state lifecycle + election logic | 1, 2, 9 | iomgr P0, vtaskmgr P0 |
| Config Bootstrap | `config_repl.c` | Pre-election probe + deferred fetch | 3 | `do_one_config_fetch` P1 |
| Startup Validation | `main.c`, `ha.c` | CLI, peer count, cert pre-flight, ASAN | 4 | argp, `access()` |
| WAL Majority-Commit | `persist.c`, `repl.c` | committed_lsn, broadcast, ack-wait | 5 | iomgr P0 |
| Persistent Control Channel | `ha.c`, `cluster.c` | TCP+SSL channel, EPOLLRDHUP, state sync | 6 | `repl.c` P0, iomgr P0 |
| Majority-Commit Config | `config_repl.c` | Config push with majority ACK | 8 | `repl_broadcast` P0 |
| Term Durability | `ha.c` | ha-term.bin persist/restore | 9 | atomic write+rename |
| Client Retry | `request.c` | Transparent retry + redirect | 7 | `XCE_ERR_*` codes P0 |
| Wire Protocol | `controller.capnp` | New opcodes and message types | 2, 3, 5, 6, 7, 8 | capnp toolchain P0 |

---

### Controller Core (ha.h / ha.c)

#### Controller FSM

| Field | Detail |
|-------|--------|
| Intent | 4-state FSM governing election, standby, and primary lifecycle; owns all election logic and term durability |
| Requirements | 1.1, 1.2, 1.3, 1.4, 2.1–2.12, 9.1–9.3 |

**Responsibilities & Constraints**
- Owns `ctrl_role_t` enum — add `CTRL_INIT=2`, `CTRL_ELECTING=3` (values above existing 0/1)
- Single allocation for `ha_elect_ctx_t` via C99 flexible array — one `calloc`, one `free`
- All role transitions via `atomic_store(&c->ha.role, ...)` — no lock held across transition
- `ha_check_priority_stepdown()` and `ha_check_heartbeat_nocluster()` are deleted in this spec
- (Note: `am_i_primary_quorum()`'s production implementation and `handle_request_vote()`/`handle_request_vote_reply()` were themselves later removed/demoted to test-only by election-iomgr-refactor — production election is now `ha_start_election_async()` → raft library; see that spec's design.md)

**New `ha_state_t` fields**:
```c
uint64_t          voted_for_term;                    // Req 9: term of last granted vote
char              voted_for_name[XCE_MAXNAME_LEN];   // Req 9: candidate of last granted vote
int               primary_ctrl_fd;                   // Req 6: persistent channel fd (-1 = none)
bool              defer_config_fetch;                // Req 3: tag=0 node defers fetch to post-election
_Atomic void     *elect_ctx;                         // Req 2: ha_elect_ctx_t* during election; NULL otherwise
_Atomic uint64_t  last_known_committed_lsn;          // Req 5: from primary heartbeats
sem_t             heartbeat_seen;                    // Req 1: posted by heartbeat handler during CTRL_INIT
uint64_t          last_known_committed_config_tag;   // Req 8: from primary heartbeats
```

**New types** (ha.h):
```c
typedef struct {
    xce_candidate_t *peer;
    uint64_t my_term;
    uint64_t my_tag;
    uint64_t my_lsn;
    int      vote_granted;
    int      replied;
    uint64_t peer_tag;
    uint64_t peer_lsn;
} ha_vote_entry_t;

typedef struct {
    int              n_peers;
    int              votes_granted;   // starts at 1 (self-vote)
    int              n_replied;
    uint64_t         max_resp_tag;
    uint64_t         max_resp_lsn;
    int              result;          // 1=primary, 0=standby, <0=error
    sem_t            done;            // startup thread blocks here; dispatch posts when done
    ha_vote_entry_t  votes[];         // C99 flexible array: n_peers-1 entries
    // Allocate: calloc(1, sizeof(*ctx) + (n_peers-1) * sizeof(ctx->votes[0]))
} ha_elect_ctx_t;
```

**`ha_persist_term()` contract**:
- Precondition: called before any vote is granted or term incremented
- Action: writes `ha_term_file_t` atomically to `{workdir}/ha-term.bin` (write + rename)
- Postcondition: file reflects current state; durable before any network reply

**Dispatch gate** (vtaskmgr switch, req 1.3):
```c
// CTRL_INIT or CTRL_ELECTING: only OP_requestVote and OP_probeConfigTag pass through
switch (atomic_load(&c->ha.role)) {
case CTRL_INIT:
case CTRL_ELECTING:
    if (msg->op != OP_requestVote && msg->op != OP_probeConfigTag) {
        xce_send_error(task, XCE_ERR_NOT_READY, "controller not ready");
        return;
    }
    break;
case CTRL_STANDBY:
    // Primary-only ops return XCE_ERR_STANDBY + redirect_host/port when known
    break;
case CTRL_PRIMARY:
    break;
}
```

**Contracts**: Service [x] / State [x]

**Implementation Notes**
- Risk: `endpoint_connect()` has a blocking TCP connect phase. Vote request sends require per-peer pthreads (not pure iomgr async). Election context `ha_elect_ctx_t` accumulates replies; threads post to `ctx->done` semaphore when quorum is reached or all peers replied. (Note: this risk was resolved by election-iomgr-refactor — vote sends are now pure async iomgr endpoints via `ha_start_election_async()`, not per-peer pthreads; `ha_elect_ctx_t` gained a refcount and several fields since, see ha-elect-ctx-lifecycle design.md)
- Integration: `ha_wait_for_heartbeat_or_timeout()` uses `sem_timedwait(&c->ha.heartbeat_seen, &deadline)` where `deadline = now + election_timeout_ms + rand_jitter`. Heartbeat handler posts `heartbeat_seen` only when `ha.role == CTRL_INIT`.
- `voted_for` enforcement: per-term, one vote per node. If `req->term > my_term`: update term, clear `voted_for_term`/`voted_for_name`. If `req->term < my_term`: deny. If already voted for a different candidate in same term: deny.

---

### Persistence Layer (persist.h / persist.c)

#### WAL Majority-Commit

| Field | Detail |
|-------|--------|
| Intent | Gate job/queue WAL writes on majority-standby durability; new committed_lsn field and truncation function |
| Requirements | 5.1–5.8 |

**New `persist_t` field**:
```c
_Atomic uint64_t committed_lsn;  // highest majority-ACKed LSN; invariant: committed_lsn <= applied_lsn
```

**`persist_truncate_wal_above(persist_t *p, uint64_t cutoff_lsn)` contract**:
- Precondition: `cutoff_lsn <= p->applied_lsn`; may be called after promotion or standby re-join
- Action: find WAL segment containing `cutoff_lsn` by filename (filename encodes start LSN as hex); scan entries in that segment sequentially until LSN > cutoff_lsn; truncate at that byte offset; delete all segments starting above cutoff_lsn; atomically update `p->applied_lsn = cutoff_lsn`
- Postcondition: no WAL entry with LSN > cutoff_lsn exists on disk; idempotent (calling twice is safe)
- Note: WAL segments are 64 MB named `seg-{LSN_hex}.wal`; no built-in LSN-to-offset index; sequential scan required

**Entry type dispatch** (5.1, 5.2):
- `WAL_JOB_SUBMIT`, `WAL_JOB_START`, `WAL_JOB_FINISH`, `WAL_QUEUE_CHANGE` → `persist_wal_sync_append()` → majority-commit path (blocking)
- `WAL_HOST_STATE`, `WAL_CHECKPOINT_REF` → `persist_wal_async()` → best-effort (unchanged)

**WAL trim guard**: trim threshold changes from `applied_lsn` to `committed_lsn` to prevent trimming uncommitted entries.

**Contracts**: Service [x] / State [x]

---

### Replication Layer (repl.h / repl.c)

#### Majority-Commit Broadcast

| Field | Detail |
|-------|--------|
| Intent | Implement repl_broadcast (was stub); add repl_wait_majority_ack; manage per-standby backpressure |
| Requirements | 5.1–5.8, 8.1–8.5 |

**New `standby_conn_t` field**: `int pending_ack_count` — backpressure counter; default max 64

**`repl_broadcast(cluster_t *c, uint8_t op, const void *payload, size_t len)` contract**:
- Sends payload to all connected standbys where `!in_catch_up && pending_ack_count <= max_pending_acks`
- Sends over persistent control channel fd (no new connection per entry)
- On send failure: remove standby from active set, trigger catch-up reconnect, log WARN
- Returns 0 when fanout initiated (not when ACKed)
- (Note: signature and mechanism superseded — current `repl_broadcast(replicator_t *r, const wal_entry_hdr_t *hdr, const void *payload, uint32_t payload_len, uint64_t prev_log_term)` sends a per-entry AppendEntries carrying the leader's per-entry term for the raft library's log-replication reconcile, with NACK-driven retry; see raft-log-replication, ha-term-safety, raft-adapter-conformance design.md)

**`repl_wait_majority_ack(cluster_t *c, uint32_t timeout_ms, uint8_t ack_op)` contract**:
- Blocks (cond_timedwait) until `acked_count + 1 >= (n_active + 1) / 2`
  - `+1` counts own write; `n_active` excludes `in_catch_up==1` standbys (5.4, Clarification 6)
- `ack_op` parameter selects `OP_walAck` vs `OP_upgradeConfigAck` to disambiguate concurrent commits
- Returns 0 on quorum, `-ETIMEDOUT` on timeout
- Single-node shortcircuit: `n_active == 0` → quorum immediately from own write (5.8, 8.5)
- (Note: signature and mechanism superseded — current `repl_wait_majority_ack(replicator_t *r, uint64_t entry_lsn, uint32_t timeout_ms, uint8_t ack_op)` delegates the walAck track to a commit engine (`commit_submit`) rather than counting acks directly, and gates commit on `durable_lsn` (fdatasync-confirmed), not raw ACK count; see ha-commit-engine and ha-commit-durability design.md)

**Contracts**: Service [x] / State [x]

---

### Config Replication (config_repl.h / config_repl.c)

#### Config Bootstrap

| Field | Detail |
|-------|--------|
| Intent | Pre-election config tag discovery and deferred post-election fetch for unconfigured nodes |
| Requirements | 3.1–3.5 |

**`bootstrap_config_if_needed(cluster_t *c)` contract**:
- Returns immediately if `c->myconf->config_version_tag != 0`
- Probes all peers via `xce_probe_config_tag()`; three outcomes:
  1. All peers `tag=0` → first-boot cluster, proceed to election (3.3)
  2. Any peer has `tag > 0` → set `c->ha.defer_config_fetch = true`, return (3.2)
  3. No peer reachable → retry with exponential backoff 500ms→10ms cap 10s; log WARN each attempt (3.4)
- Never returns an error; blocks until resolution or process killed
- (Note: this full contract was later simplified/removed — the production `bootstrap_config_if_needed()` was removed in election-iomgr-refactor task 5.3; only a stub remains for linkage, mirroring the simplified two-branch logic now inlined ahead of `ha_start_election_async()` in myapp.c. See election-iomgr-refactor design.md)

**`xce_probe_config_tag(const char *host, uint16_t port, uint64_t *tag_out)` contract**:
- Single-RTT SSL request via `connect_ssl_raw()` + xce encode/decode
- Request: `OP_probeConfigTag` with `ProbeConfigTagRequest{name}` (requester name for peer logging)
- Reply: `ProbeConfigTagReply{config_tag}` (0 if peer has no config)
- Peer answers in any FSM state — dispatch gate exempts `OP_probeConfigTag` alongside `OP_requestVote`
- Returns 0 on success, `-ECONNREFUSED` / `-ETIMEDOUT` on unreachable peer

#### Majority-Commit Config Push

| Field | Detail |
|-------|--------|
| Intent | Upgrade push_config_to_standbys() from fire-and-forget to majority-commit using shared repl infrastructure |
| Requirements | 8.1–8.5 |

**`push_config_to_standbys()` extended contract**:
1. Write config blob locally (existing atomic write+rename — unchanged)
2. `repl_broadcast(c, OP_upgradeConfig, blob, len)`
3. `repl_wait_majority_ack(c, commit_timeout_ms, OP_upgradeConfigAck)`
4. On quorum: `atomic_store(&c->myconf->committed_config_tag, tag)` (8.1)
5. On timeout: return `-ETIMEDOUT`; operator retries (8.3)

**`committed_config_tag`**: new field in cluster config state; piggybacked on `CtrlHeartbeat` (`committed_config_tag @N :UInt64`). Standbys store in `ha.last_known_committed_config_tag`. On promotion: serve only configs up to `last_known_committed_config_tag` (8.4).

**Contracts**: Service [x] / State [x]

---

### Persistent Control Channel (ha.c / cluster.c)

#### Primary Discovery and Control Channel

| Field | Detail |
|-------|--------|
| Intent | Persistent TLS channel per standby; EPOLLRDHUP failover; full state sync on join |
| Requirements | 6.1–6.8 |

**Three-tier primary discovery** (post-election, standby path):
1. **Tier 1** (O(1) for majority): if `ha.voted_for_name` set, try direct connect to voted-for peer; on `XCE_ERR_NOT_READY` peer still electing — backoff, retry Tier 1
2. **Tier 2** (redirect): probe remaining peers; a `CTRL_STANDBY` peer that knows the primary replies with `XCE_ERR_STANDBY` + `redirect_host/port`; follow redirect immediately
3. **Tier 3** (blind scan): try remaining peers sequentially with 100ms→5s exponential backoff until primary found

**Extended `OP_ctrlHello` schema** (controller.capnp):
```
struct CtrlHelloRequest {
  name        @0 :Text;
  key         @1 :Data;    # SHA-256(cluster_name + shared_secret); auth gate
  applied_lsn @2 :UInt64;
  config_tag  @3 :UInt64;
}
struct CtrlHelloReply {
  committed_lsn  @0 :UInt64;
  needs_snapshot @1 :Bool;
  config_tag     @2 :UInt64;
}
```

**Failover detection** — dual-trigger replacing timestamp polling (6.2, 6.3, 6.4):
- **Fast**: `EPOLLRDHUP` on `primary_ctrl_fd` → immediately close, null fd, set `CTRL_ELECTING`
- **Fallback**: heartbeat timeout vtask armed on each heartbeat receipt; fires after `election_timeout_ms + rand_jitter` of silence; same action as EPOLLRDHUP path

**`XCE_ERR_STANDBY` redirect extension** (req 7.2, Tier 2):
Add to `RequestReply` (controller.capnp):
```
redirect_host @N :Text;    # empty string when primary unknown
redirect_port @M :UInt16;  # 0 when primary unknown
```
Standbys populate from `ha.primary_host/port` when known; clients without redirect fields fall back to normal retry.

**State sync sequence on standby join** (6.5, 6.6, 6.7):

Primary side (`handle_ctrlhello()`):
1. `repl_add_standby()` → `repl_start_catchup()` sets `in_catch_up=1`
2. Reply `CtrlHelloReply{committed_lsn, needs_snapshot, config_tag}`
3. If `needs_snapshot`: `persist_checkpoint()` → send `OP_snapshotTransfer`
4. Stream `OP_walCatchupEntry` [applied_lsn+1 → current committed_lsn]
5. On `OP_walAck` reaching committed_lsn → `repl_complete_catchup()` (`in_catch_up=0`)
6. If standby config_tag ≠ primary config_tag → `push_config_to_standbys()`

Standby side:
1. Store `primary_committed_lsn = reply.committed_lsn`; update `ha.last_known_committed_lsn`
2. `persist_truncate_wal_above(primary_committed_lsn)` — discard orphaned uncommitted entries
3. `ha.in_catch_up = 1` — deny votes during sync (Clarification 6)
4. If `reply.needs_snapshot`: receive `OP_snapshotTransfer`, write, call `load_snapshot()`
5. Apply each `OP_walCatchupEntry` via `apply_wal_entry()`; send `OP_walAck` after each
6. On catch-up loop end from primary: `ha.in_catch_up = 0`

**Role flip on promotion** (6.8): standby closes and nulls `primary_ctrl_fd` before returning from `am_i_primary_quorum()` (Note: this call site moved to the raft library's `on_become_leader` callback path, `raft_cb_become_leader`, since `am_i_primary_quorum()` was removed — see election-iomgr-refactor); new primary accepts inbound `OP_ctrlHello` connections in a per-standby fd table.

**Contracts**: Service [x] / State [x]

**Implementation Notes**
- Risk: `apply_wal_entry()` must exist before Change F lands; extract from startup WAL replay path as a prerequisite task
- Risk: `persist_truncate_wal_above()` requires sequential scan of partial WAL segment; validate with unit tests before integrating into standby join path
- Integration: `repl_broadcast()` skips `in_catch_up==1` standbys during the streaming loop; `repl_complete_catchup()` fires only when standby's `acked_lsn` equals current `committed_lsn`

---

### Client Retry (src/lib/request.c)

**(Note: superseded by libxce-failover)** — `xce_request_with_retry()` described below was later removed as an "orphaned no-op" (per `src/lib/request.c` comment at the definition of `XCE_RETRY_DEFAULTS`); it was superseded by a `StatusClassifier`/`EndpointRotator` failover loop in `src/lib/client_failover.c`, driven from the xcecli layer. `xce_retry_config_t` and `XCE_RETRY_DEFAULTS` (the tunable struct/defaults — max_retries=20, initial_retry_ms=50, max_retry_ms=5000, backoff_multiplier=2.0) still exist verbatim in `src/include/request.h` and are consumed by the new failover loop. See libxce-failover design.md for the current mechanism.

#### libxce Retry Wrapper

| Field | Detail |
|-------|--------|
| Intent | Transparent retry for transient controller errors; existing daemon code unchanged |
| Requirements | 7.1–7.6 |

**`xce_retry_config_t`**:
```c
typedef struct {
    int    max_retries;         // default 20
    int    initial_retry_ms;    // default 50
    int    max_retry_ms;        // default 5000
    double backoff_multiplier;  // default 2.0 (exponential)
} xce_retry_config_t;
```

**`xce_request_with_retry(endpoint_t *ep, xce_request_t *req, const xce_retry_config_t *cfg)` contract**:
- NULL `cfg` → uses defaults
- `XCE_ERR_NOT_READY`: backoff, retry same endpoint (7.1)
- `XCE_ERR_STANDBY` with `redirect_host/port`: reconnect to redirect, retry (7.2)
- `XCE_ERR_STANDBY` without redirect: backoff, retry same endpoint (7.2 fallback)
- `ECONNREFUSED` / `ETIMEDOUT`: reconnect, retry (7.3)
- `XCE_ERR_AUTH`, `XCE_ERR_PERM`, `XCE_ERR_INVALID`, `XCE_ERR_NOENT`: surface immediately (7.4)
- Exhausted `max_retries`: surface last error to caller

**Contracts**: Service [x]

**Implementation Notes**
- Current `xce_request()` is async send-only; the retry wrapper handles the synchronous request-reply cycle around it
- Backoff jitter uses simple `rand_r()` to avoid thundering-herd (7.6)

---

### Wire Protocol (src/schema/controller.capnp)

#### New Opcodes and Message Types

Append after existing `upgradeConfig @31`:

| Opcode | @N | Purpose | Req |
|--------|----|---------|-----|
| `requestVote` | @32 | Raft vote request/reply | 2 |
| `probeConfigTag` | @33 | Lightweight config tag probe | 3 |
| `walAck` | @34 | WAL majority-commit ACK | 5 |
| `upgradeConfigAck` | @35 | Config push majority-commit ACK | 8 |
| `snapshotTransfer` | @36 | Snapshot blob from primary to standby | 6 |
| `walCatchupEntry` | @37 | WAL entry during catch-up streaming | 6 |

**Key struct definitions** (to add to controller.capnp):
```
struct RequestVoteRequest {
  term         @0 :UInt64;
  candidateId  @1 :Text;
  configTag    @2 :UInt64;
  lastLogLsn   @3 :UInt64;
}
struct RequestVoteReply {
  term             @0 :UInt64;
  voteGranted      @1 :Bool;
  peerConfigTag    @2 :UInt64;
  peerLastLogLsn   @3 :UInt64;
}
struct ProbeConfigTagRequest { name @0 :Text; }
struct ProbeConfigTagReply   { configTag @0 :UInt64; }
struct WalAckRequest  { term @0 :UInt64; lsn @1 :UInt64; }
struct WalAckReply    { term @0 :UInt64; lsn @1 :UInt64; success @2 :Bool; }
struct UpgradeConfigAck { term @0 :UInt64; ackedTag @1 :UInt64; success @2 :Bool; }
struct SnapshotTransfer { snapshotLsn @0 :UInt64; payload @1 :Data; }
struct WalCatchupEntry  { lsn @0 :UInt64; entryType @1 :UInt8; payload @2 :Data; }
```
(Note: `WalCatchupEntry.payload` is now `List(UInt8)`, not `Data` — bug-1505 capnpc-c codec fix, unrelated to any listed spec)

Also extend:
- `CtrlHeartbeat`: add `committedLsn @N :UInt64` and `committedConfigTag @M :UInt64` for piggybacking (req 5.6, 8.4)
- `RequestReply`: add `redirectHost @N :Text` and `redirectPort @M :UInt16` for standby redirect (req 7.2)

---

## Data Models

### New Persistent State

**`ha-term.bin`** — fixed-size binary, atomic write+rename:
```c
typedef struct {
    uint64_t magic;                       // 0xDEADBEEFCAFEBABE
    uint64_t term;
    uint64_t voted_for_term;
    char     voted_for_name[XCE_MAXNAME_LEN];
    uint32_t crc32;                       // CRC32 of all preceding fields
} ha_term_file_t;
```
- Read in `ha_init()` at startup; missing file → term=0, no prior vote (first boot)
- CRC mismatch → log WARN, initialize to term=0 (conservative: safe to vote again)

### New In-Memory State

| Struct | Field | Type | Invariant |
|--------|-------|------|-----------|
| `persist_t` | `committed_lsn` | `_Atomic uint64_t` | `committed_lsn <= applied_lsn` always |
| `ha_state_t` | 8 new fields | various | See Components section |
| `standby_conn_t` | `pending_ack_count` | `int` | `<= max_pending_acks` while in active set |

### Consistency Guarantees

- WAL trim guard: threshold = `committed_lsn` not `applied_lsn`; prevents trimming entries a promoted standby may need for divergence resolution
- `committed_lsn <= applied_lsn` asserted at promotion time; violated invariant is a programming error
- `ha-term.bin` is written atomically before any vote reply is sent; a crash between write and reply means the term is persisted but the vote may not have been counted — conservative and safe
- (Note: majority-commit via `committed_lsn` alone is superseded — ha-commit-durability added a `durable_lsn` field: a member counts toward commit majority only once its LSN is `fdatasync`-confirmed, not merely ACKed/appended; `committed_lsn` still exists but the true commit/promotion-safety floor is `persist_durable_floor()`. See ha-commit-durability design.md)

---

## Error Handling

### Election Errors

| Error | Condition | Response |
|-------|-----------|----------|
| `config_tag=0` and any peer has `tag > 0` | Hard gate A | Fatal exit `-ENODATA`; log ERROR with peer's tag value |
| Quorum not reached after 2 rounds | Normal election result | Enter `CTRL_STANDBY`; log WARN with vote tally |
| Fresher responding peer yield gate B | Winner yields to peer with higher tag or LSN | Enter `CTRL_STANDBY`; log INFO with peer tag/lsn seen |
| `ha-term.bin` CRC mismatch at startup | File corruption or incomplete write | Log WARN; initialize term=0; safe to proceed |
| Split vote after 2nd round | Split-vote safety net | Enter `CTRL_STANDBY`; log WARN |

### WAL and Config Commit Errors

| Error | Condition | Response |
|-------|-----------|----------|
| `-ETIMEDOUT` from `repl_wait_majority_ack` | No quorum within `commit_timeout_ms` | Return to caller; entry persisted locally but not committed; log WARN |
| Standby backpressure exceeded | `pending_ack_count > max_pending_acks` | Remove from active set; trigger catch-up reconnect; log WARN |
| Standby send failure in `repl_broadcast` | TCP error on control channel fd | Same as backpressure — remove and trigger reconnect |

### Startup Validation Errors

| Error | Condition | Response |
|-------|-----------|----------|
| Even peer count | `n % 2 == 0 && n > 1` | Fatal `-EINVAL`; operator-friendly stderr with correct format |
| Cert missing or unreadable | `access(path, R_OK) != 0` | Fatal `-ENOENT`; print full path |
| `--vcpus < 1` | CLI argument | Fatal `-EINVAL`; print message |
| `vcpus` key in config file | Deprecation | Log WARN; do not fail startup |

### Operational Constraint — WAL Stall Window

Between primary winning election and first standby completing `repl_complete_catchup()`, the active replication set has 0 standbys. Every `persist_wal_sync_append()` call blocks for `commit_timeout_ms` and returns `-ETIMEDOUT`. This is correct CP behavior. Libxce retry (req 7) handles this transparently for workers and schedulers. Operators must be warned in deployment documentation.

### Monitoring (Req 10)

All new paths emit structured `xcelog.h` log entries:
```c
// Election completion (10.1)
INFO("election complete: role=%s term=%lu votes=%d/%d quorum=%d",
     role_str, term, granted, n_peers, quorum);
// Bootstrap probe result (10.2)
INFO("config_bootstrap: outcome=defer best_peer_tag=%016llx", best_tag);
// WAL commit quorum (10.3)
INFO("wal_commit: lsn=%lu acked=%d/%d latency_ms=%d", lsn, acked, n_active, ms);
// Config push quorum (10.4)
WARN("config_push_timeout: tag=%016llx timeout_ms=%d", tag, commit_timeout_ms);
// FSM transition (10.5)
INFO("ha_role: %s -> %s term=%lu", from_str, to_str, term);
```

---

## Testing Strategy

### Unit Tests (`src/controller/testsuite/`)

| File | Verifies | Req |
|------|---------|-----|
| `test_ha_election.c` | Quorum met on 2/3 votes; voted_for enforcement across terms; freshness gate yields to fresher peer; split-vote triggers retry; single-node self-elects immediately | 2.1–2.12 |
| `test_ha_bootstrap.c` | All-zero first-boot proceeds; peer-has-config sets defer_config_fetch; no-peer-reachable retries with backoff; retry succeeds on 3rd attempt | 3.1–3.5 |
| `test_ha_durability.c` | ha_persist_term writes then ha_init restores term and voted_for; CRC corruption zeroes state safely; missing file zeroes state | 9.1–9.3 |
| `test_persist_truncate.c` | Truncation at segment boundary deletes whole segment; truncation mid-segment scans to correct offset; calling twice with same cutoff is idempotent | 5.6, 5.7 |
| `test_repl_majority.c` | Majority reached on 2/3 ACKs; timeout returns -ETIMEDOUT; single-node returns 0 immediately; in_catch_up standby excluded from quorum count | 5.1–5.8, 8.1–8.5 |
| `test_ha_startup.c` | Even peer count rejected; cert missing returns -ENOENT with path; --vcpus 0 rejected; vcpus in config logs deprecation warning but does not fail | 4.1–4.4 |

### Unit Tests (`src/lib/testsuite/`)

| File | Verifies | Req |
|------|---------|-----|
| `test_libxce_retry.c` | NOT_READY triggers backoff and retry same host; STANDBY with redirect connects to redirected host; STANDBY without redirect falls back to backoff; AUTH error surfaces immediately; max_retries exhausted surfaces last error; 50 concurrent clients with backoff do not starve | 7.1–7.6 |

### Integration Tests (multi-node, `make test-integration`)

| Scenario | Verifies | Req |
|----------|---------|-----|
| Concurrent-startup: both nodes start within 3s | Only one node wins election; no false self-election | 1.1, 2.1 |
| Config-fresh node vs stale node election | Stale node yields; fresh node wins | 2.7 |
| Unconfigured node joins live cluster | defer_config_fetch set; deferred fetch succeeds; node enters standby with correct config | 3.2, 3.5 |
| Primary crash → standby promotion | Sub-second EPOLLRDHUP detection; promoted standby has all committed WAL entries | 5.6, 6.2 |
| WAL commit with all standbys down | Returns -ETIMEDOUT per entry; single-standby quorum resumes on reconnect | 5.5 |
| Config upgrade under network partition | Operator receives -ETIMEDOUT; retry after heal succeeds | 8.3 |
| Restart after participating in term N | Restored voted_for prevents double-vote | 9.1, 9.2 |

### Performance Tests

| Test | Target | Req |
|------|--------|-----|
| Election round-trip latency (3-node LAN) | < 100ms p99 | 2.1 |
| WAL majority-commit overhead (3-node LAN) | < 5ms p99 per WAL_JOB_SUBMIT | 5.1 |
| Failover detection latency | < 1s from TCP disconnect to election start | 6.2 |
| 50 concurrent libxce clients post-failover | All succeed within 30s; no starvation | 7.6 |

---

## Security Considerations

- **Cert pre-flight** (4.2): Controller refuses to start with missing or unreadable TLS cert/key. Prevents silent unencrypted operation.
- **Cluster auth key in `OP_ctrlHello`** (6.1): `key = SHA-256(cluster_name + shared_secret)` gates standby join. `cluster_key` is a new field read from the existing config file (not CLI, to avoid shell history exposure). Primary rejects `OP_ctrlHello` where HMAC does not match.
- **`voted_for` persistence** (9.1): Prevents double-vote within a term across restarts. A node that crashes after voting and restarts must not vote for a second candidate in the same term.
- **Term-based fencing**: A primary that receives any reply with `term > my_term` immediately steps down (`ha_do_demote()`). Prevents a reconnected stale primary from continuing after partition heal.
- **No priority tiebreak**: Raft freshness (config_tag + LSN) fully determines winner. The `ha.my_priority` field is preserved in `ha_state_t` for backward compatibility but plays no role in election outcome.

## Performance & Scalability

- **Write latency**: 1 LAN RTT + max(fsync_primary, fsync_standby) per majority-commit entry. At 0.1–0.5ms LAN RTT and 0.05–0.2ms NVMe: ~0.2–0.7ms commit latency. Applies only to `WAL_JOB_SUBMIT`, `WAL_JOB_START`, `WAL_JOB_FINISH`, `WAL_QUEUE_CHANGE`.
- **Throughput ceiling**: 1,000–20,000 job submits/sec before jobmgr locking dominates. Raft overhead is not the bottleneck at HPC-typical rates.
- **5-node vs 3-node write latency**: negligible. 5-node waits for 2nd-fastest of 4 standbys vs 1st of 2; sub-millisecond difference on uniform LAN.
- **Single-node shortcircuit**: all majority-commit paths check `n_peers == 1` first; self-only write immediately satisfies quorum.
- **Backpressure**: `max_pending_acks = 64` (configurable) prevents any single slow standby from stalling primary fanout.
- **`commit_timeout_ms`**: configurable, default 5000ms; tune per deployment network latency.