# Brief: ha-election-raft-config

## Problem

The current LSF-style priority-based election has three critical safety gaps:

1. **Config freshness not guaranteed**: A node with missing or stale config can be elected primary and serve incorrect cluster state to workers/schedulers.

2. **False self-elections during concurrent startup**: When pc13v1 starts while pc13v (the priority-0 primary) is still initializing, pc13v's peer listener port is not open yet. pc13v1's election probe gets ECONNREFUSED, falsely concludes the primary is absent, and self-elects. Both nodes declare themselves primary.

3. **No bootstrap path for genuine failover**: When a standby is promoted during real failover (original primary down), it has no mechanism to pull config from peer standbys if its local `conf/current` is missing.

Reproduction from logs:
- pc13v starts at 12:30:03, pc13v1 starts at 12:30:22 (19s later)
- pc13v1 probes pc13v:9990 during election, times out within 3046ms
- pc13v is healthy but still in `load_primary_config()` or early phase2 — listener not open
- pc13v1 wins: "no higher-priority candidate responded"
- pc13v1 calls `load_primary_config()` → "Primary has no config at '/tmp/xce/pc13v1/conf/current' (rc=-2)" → fatal exit

## Current State

- **Election** (ha.c:243–306): LSF-style priority-based. Priority-0 always wins unconditionally; priority-N probes [0..N-1] and self-elects on timeout. No quorum. No config-version check. Logically correct for simple cases but unsafe for config freshness.
- **Listener timing** (myapp.c:21–96): `bind()+listen()` happens inside `cluster_init_phase2()` AFTER election and config load. Creates a 2–3 second window where the primary is unreachable to probes.
- **Config versioning** (config_repl.c): SHA-256 tagged, atomic writes, retention policy, staleness detection (`conf_stale` flag). Complete and correct.
- **Config fetch** (config_repl.c:891–965): standby→primary pull framework exists; `do_one_config_fetch()` returns `-ENOSYS` (stub, Task 11 on branch).
- **Bootstrap after failover**: not implemented. Design §14.4 deferred quorum consensus.
- **Simultaneous primaries**: `ha_check_priority_stepdown()` (ha.c:322) has a term=0 exact check that fails to resolve races when both nodes increment term before heartbeat exchange.

## Availability Trade-off (CP chosen)

This design chooses **Consistency over Availability** (CP in CAP terms):

- A cluster of N nodes requires a **majority (≥(N+1)/2)** for both **election** and **write commits**.
- **Election**: A minority partition cannot elect a primary. The cluster is unavailable until the majority partition is restored.
- **Writes (WAL replication)**: Primary waits for majority standby ACKs before confirming a write (job submit, queue change, state update). In-flight writes that cannot reach majority are rejected. This adds **~1 LAN RTT per write** (~0.1–5ms depending on network).
- **Zero data loss on failover**: A promoted standby is guaranteed to have all committed WAL entries because majority replication ensures durability before write ACK.
- **Operational implication**: Plan for N≥3 nodes. A 2-node cluster has no meaningful majority; it is equivalent to single-node with a cold standby. For HPC, zero data loss (no lost job submissions, no stale queue state) outweighs the write-latency cost.

## Desired Outcome

1. **Raft-style quorum election (CP)**: A candidate must obtain ACKs from a strict majority of peers (≥(N+1)/2) before becoming primary. A minority partition cannot elect a primary. The cluster is unavailable until the majority partition is restored.

2. **Config freshness encoded in votes**: Each vote reply includes the peer's config tag and WAL LSN. The candidate with the freshest config (highest tag, then highest LSN) among quorum members wins tiebreaks.

3. **No-config nodes bootstrap after election (deferred fetch)**: A node with `config_tag=0`
   never initiates candidacy. It probes peers to determine whether config exists anywhere,
   but **defers the actual fetch until the primary is known** — the election proves who
   has the freshest state, so bootstrap fetches from the proven winner rather than
   speculatively guessing. Three sub-cases:
   - `tag=0`, peers have config → set `defer_config_fetch`, participate as voter only, fetch from winner after election
   - `tag=0`, all peers also `tag=0` → first-boot cluster, proceed to election as-is
   - `tag=0`, no peer reachable → retry: probe loop with exponential backoff (500ms→10s);
     log WARN every attempt; block indefinitely until a peer responds or admin kills

4. **Listener before election**: Peer listener socket opens before election so concurrent-startup probes reach an open port. False self-elections during overlapping startup are eliminated.

5. **Config freshness preserved through all paths**: A node always has a valid config
   before entering phase 2 — either from local disk (normal case) or fetched from the
   proven election winner after the election resolves (no-config case, deferred fetch in
   Change C). A quorum winner therefore always has `config_tag > 0` when peers do, and
   serves correct cluster state immediately after promotion with no additional bootstrap step.

6. **Most up-to-date primary**: Among all reachable nodes, only the one with the
   highest config_tag (then highest LSN) can become primary. A candidate that wins
   the vote count but sees a fresher responding peer yields back to standby —
   the fresher peer runs its own election and wins. Non-responding (partitioned or
   crashed) peers never block election.

7. **No simultaneous primaries**: Only one node can obtain a quorum at any given term. Term-based fencing prevents split-brain on partition heal. `voted_for` tracking (persisted to `ha-term.bin`) enforces one vote per term per node across restarts.

8. **Majority-commit WAL replication (CP writes)**: Primary waits for majority (≥(N+1)/2) standby ACKs before marking a WAL entry committed and confirming the write to the caller. In-flight writes that cannot reach majority within a timeout are rejected. Zero data loss on failover: promoted standby has all committed entries.

9. **Majority-commit config push (CP config)**: Config upgrades (`OP_upgradeConfig`) use the
   same `repl_broadcast()` + `repl_wait_majority_ack()` infrastructure as WAL replication —
   the primary waits for ≥(N+1)/2 standby ACKs before confirming a config change to the
   operator. Operator changes cannot be silently lost on failover.

10. **Hard SSL/cert gate**: Controller refuses to boot if TLS cert/key files are missing or unreadable. Operator-friendly error message with file path.

11. **Observable via logs**: Election results, vote tallies, quorum counts, config tag comparisons, bootstrap operations, WAL commit quorum status, config push quorum status, and cert validation logged at INFO or WARN severity.

## Approach

**Replace LSF priority election with Raft-style quorum voting AND add majority-commit WAL replication for full CP durability. Config upgrades use the same majority-commit infrastructure for uniform CP semantics.**

Eight focused changes:

### Change A — Move listener, iomgr, and vtaskmgr before election

Extract `bind()+listen()`, `iomgr_init()`, and `vtaskmgr_init()` from
`cluster_init_phase2()` so the full connection framework is running before election.
`vtaskmgr_init()` uses the new `--vcpus N` CLI parameter (Change D item 1) instead of
`cluster_config->vcpus`, removing the last config dependency on this startup block.

New startup order in `myapp.c`:
```
validate_peer_count()          ← Change D: odd-N check before any init
validate_certs()               ← Change D: cert pre-flight
cluster_init_phase1()          ← ha_init() reads ha-term.bin: restores term + voted_for
                               ← conf_read_tag() reads config_version_tag from disk (0 if absent)
                               ← Node now has durable identity before network is up
create_ssl_context()
iomgr_init() + iomgr_set_ssl_ctx()
cluster_open_listener(port)    ← bind + listen; register fd with iomgr
vtaskmgr_init(vcpus)
iomgr_start()
vtaskmgr_start()
atomic_store(ha.role, CTRL_INIT)    ← node now answers OP_requestVote + OP_probeConfigTag
bootstrap_config_if_needed()  ← Change C: if config_tag==0, probe peers; defers fetch until primary known
ha_wait_for_heartbeat_or_timeout()  ← Raft: sem_timedwait(election_timeout + rand_jitter)
                               ← heartbeat received → CTRL_STANDBY, skip election
                               ← timeout fires   → CTRL_ELECTING → am_i_primary_quorum()
[primary or standby path]
load_primary_config() / ha_connect_to_primary()
cluster_init_phase2_rest()    ← kvmgr, xstring, jobmgr, delaymgr, apiserver, etc.
cluster_run()
```

With iomgr and vtaskmgr running before election, incoming `OP_requestVote` messages
are handled by the normal dispatch switch — no special pre-framework path needed.
Outgoing vote requests use the existing `connect_ssl_raw()` + `xce_encode()/xce_decode()`
pattern (same as config fetch), keeping per-peer threads lightweight.

**Full 4-state controller FSM (`ha_role_t`)**:

The controller lifecycle is a single FSM stored in `c->ha.role` (atomic):
```
CTRL_INIT ──── timeout + jitter ──→ CTRL_ELECTING ──── quorum won ──→ CTRL_PRIMARY
          └─── heartbeat from        │                                     │
               existing primary ─→  └─── quorum lost ─→ CTRL_STANDBY ────┘
                                                              │
                                          EPOLLRDHUP / heartbeat timeout
                                                              │
                                                         CTRL_ELECTING
```

**Dispatch gate** (top of vtaskmgr switch, before any `case`):
```c
switch (atomic_load(&c->ha.role)) {
case CTRL_INIT:
case CTRL_ELECTING:
    // Only vote + config-probe pass; all others get NOT_READY and retry (Change G)
    if (msg->op != OP_requestVote && msg->op != OP_probeConfigTag) {
        xce_send_error(task, XCE_ERR_NOT_READY, "controller not ready");
        return;
    }
    break;
case CTRL_STANDBY:
    // Primary-only ops rejected with XCE_ERR_STANDBY + redirect_host/port (Change G)
    break;
case CTRL_PRIMARY:
    break;  // full dispatch
}
```

**State transition triggers**:
- `CTRL_INIT → CTRL_ELECTING`: `ha_wait_for_heartbeat_or_timeout()` times out — no
  heartbeat received within `election_timeout + rand_jitter`. Different jitter per node
  (xorshift64 from `jitter_seed`) means nodes fire at staggered times, so the first
  candidate typically wins unopposed. This is Raft's primary split-vote prevention.
- `CTRL_INIT → CTRL_STANDBY`: heartbeat or `OP_ctrlHello` received from an existing
  primary during the wait window — node skips election entirely.
- `CTRL_ELECTING → CTRL_PRIMARY`: `am_i_primary_quorum()` returns 1.
- `CTRL_ELECTING → CTRL_STANDBY`: `am_i_primary_quorum()` returns 0.
- `CTRL_STANDBY → CTRL_ELECTING` (fast): `EPOLLRDHUP` on `primary_ctrl_fd`.
- `CTRL_STANDBY → CTRL_ELECTING` (fallback): heartbeat timeout vtask fires — primary
  is alive on TCP but not sending heartbeats. Vtask is reset on each heartbeat received;
  fires after `election_timeout + rand_jitter` of silence.

**`ha_wait_for_heartbeat_or_timeout()`** in `myapp.c`:
```c
struct timespec dl = now_plus_ms(c->ha.config.election_timeout_ms +
                                  ha_rand_jitter(&c->ha));
int hb = (sem_timedwait(&c->ha.heartbeat_seen, &dl) == 0);
if (hb) {
    atomic_store(&c->ha.role, CTRL_STANDBY);
} else {
    atomic_store(&c->ha.role, CTRL_ELECTING);
    rc = am_i_primary_quorum(c);
}
```
`ha.heartbeat_seen` is posted by the heartbeat dispatch handler when any heartbeat
or `OP_ctrlHello` is received while `ha.role == CTRL_INIT`.

### Change B — Raft-style quorum voting in `am_i_primary()`

Replace the current timeout-based probe with a full request-vote protocol:

**New wire message: OP_requestVote** (Cap'n Proto)
```c
struct RequestVoteRequest {
  term @0 :UInt64;              // candidate's term
  candidate_id @1 :Text;        // candidate's name (from --peers)
  config_tag @2 :UInt64;        // candidate's config SHA-256 tag (0 if none)
  last_log_lsn @3 :UInt64;      // candidate's WAL applied_lsn
  // NOTE: priority removed — Raft config_tag+LSN freshness fully determines winner;
  // jitter breaks split votes. Priority plays no role in quorum election.
}

struct RequestVoteReply {
  term @0 :UInt64;              // peer's current term
  vote_granted @1 :Bool;        // true = ACK
  peer_config_tag @2 :UInt64;   // peer's config tag (for config comparison)
  peer_last_log_lsn @3 :UInt64; // peer's WAL LSN (for log comparison)
}
```

**Election uses iomgr + vtaskmgr (FSM model)**:

Since iomgr and vtaskmgr are running before election (Change A), vote requests are
sent via iomgr's async connect path and replies arrive through the normal vtaskmgr
dispatch switch. No election-specific threads are created.

**New `ha_vote_entry_t`** struct (ha.h) — per-peer election slice; only meaningful as
`ctx->votes[i]` inside `ha_elect_ctx_t`:
```c
typedef struct {
    xce_candidate_t *peer;
    uint64_t my_term;       // term sent in RequestVote
    uint64_t my_tag;        // config_tag sent in RequestVote
    uint64_t my_lsn;        // WAL LSN sent in RequestVote
    int      vote_granted;  // 1 = peer granted vote
    int      replied;       // 1 = peer responded (any reply, including denial)
    uint64_t peer_tag;      // peer's config_version_tag from reply
    uint64_t peer_lsn;      // peer's applied_lsn from reply
    // NOTE: no pthread_t — FSM uses iomgr async sends + vtaskmgr dispatch;
    //       ha_elect_ctx_t.done semaphore blocks am_i_primary_quorum()
} ha_vote_entry_t;
```

**New `ha_elect_ctx_t`** struct (ha.h) — holds all election state; single allocation via
C99 flexible array member (one `calloc`, one `free`):
```c
typedef struct {
    int              n_peers;
    int              votes_granted;  // includes self-vote (starts at 1)
    int              n_replied;      // peers that responded (any reply)
    uint64_t         max_resp_tag;   // freshest config_tag seen across all replies
    uint64_t         max_resp_lsn;   // freshest lsn seen across all replies
    int              result;         // set before sem_post: 1=primary, 0=standby, <0=error
    sem_t            done;           // startup thread blocks here; dispatch posts when done
    ha_vote_entry_t  votes[];        // C99 flexible array — n_peers-1 entries
} ha_elect_ctx_t;
// Allocate: calloc(1, sizeof(*ctx) + (n_peers-1)*sizeof(ctx->votes[0]))
// Free:     free(ctx)  — single allocation, no separate votes free needed
```

**`am_i_primary_quorum()`** (ha.c) — initiates election, blocks until done:
```c
int am_i_primary_quorum(cluster_t *c) {
    int n_peers = c->pcfg.n_candidates;
    if (n_peers == 1) return 1;

    uint64_t my_term = atomic_fetch_add(&c->ha.term, 1) + 1;
    ha_persist_term(c, my_term, c->myconf->name);

    ha_elect_ctx_t *ctx = calloc(1, sizeof(*ctx) + (n_peers-1)*sizeof(ctx->votes[0]));
    ctx->n_peers       = n_peers;
    ctx->votes_granted = 1;   // self-vote
    ctx->max_resp_tag  = c->myconf->config_version_tag;
    ctx->max_resp_lsn  = atomic_load(&c->persist.applied_lsn);
    sem_init(&ctx->done, 0, 0);
    atomic_store(&c->ha.elect_ctx, ctx);  // dispatch reads this

    // Send RequestVote to all peers via iomgr (async non-blocking connects)
    for (int i = 0, j = 0; i < n_peers; i++) {
        if (strcmp(c->pcfg.candidates[i].name, c->myconf->name) == 0) continue;
        ctx->votes[j].peer   = &c->pcfg.candidates[i];
        ctx->votes[j].my_term = my_term;
        ctx->votes[j].my_tag  = c->myconf->config_version_tag;
        ctx->votes[j].my_lsn  = atomic_load(&c->persist.applied_lsn);
        iomgr_async_vote_request(c, &ctx->votes[j]);  // non-blocking
        j++;
    }

    // Arm election timeout vtask — fires ha_elect_timeout() if peers don't reply
    vtaskmgr_schedule(c, c->ha.config.election_timeout_ms, ha_elect_timeout, ctx);

    sem_wait(&ctx->done);  // startup blocks here; dispatch or timeout signals it

    int result = ctx->result;
    ha_elect_ctx_free(ctx);
    atomic_store(&c->ha.elect_ctx, NULL);
    return result;
}
```

**`handle_request_vote_reply()`** — called from vtaskmgr dispatch switch on
`OP_requestVoteReply`; accumulates votes and signals completion:
```c
void handle_request_vote_reply(cluster_t *c, RequestVoteReply *reply, int peer_idx) {
    ha_elect_ctx_t *ctx = atomic_load(&c->ha.elect_ctx);
    if (!ctx) return;  // no election in progress

    ha_vote_entry_t *v = &ctx->votes[peer_idx];
    v->replied = 1;
    v->peer_tag = reply->peer_config_tag;
    v->peer_lsn = reply->peer_last_log_lsn;
    if (reply->vote_granted) { v->vote_granted = 1; ctx->votes_granted++; }
    if (v->peer_tag > ctx->max_resp_tag ||
        (v->peer_tag == ctx->max_resp_tag && v->peer_lsn > ctx->max_resp_lsn)) {
        ctx->max_resp_tag = v->peer_tag; ctx->max_resp_lsn = v->peer_lsn;
    }
    ctx->n_replied++;

    int quorum = (ctx->n_peers + 1) / 2;
    // Signal when quorum achieved or all peers replied (or timed out)
    if (ctx->votes_granted >= quorum || ctx->n_replied == ctx->n_peers - 1)
        ha_elect_evaluate_and_signal(c, ctx);
}
```

**`ha_elect_timeout()`** vtask — fires when `election_timeout_ms` expires:
counts unreplied peers as non-respondents, calls `ha_elect_evaluate_and_signal()`.

**`ha_elect_evaluate_and_signal()`** — applies gates, sets `ctx->result`, signals:
- Defense-in-depth assertion (config_tag=0 BUG check)
- Hard gate B (yield if responding peer has fresher data → `result=0`)
- Quorum met → `result=1`
- Quorum lost → schedule jitter vtask, retry once with incremented term;
  if second round also fails → `result=0`; finally `sem_post(&ctx->done)`

**Vote handler on peers** (new function in ha.c):
```c
void handle_request_vote(cluster_t *c, RequestVoteRequest *req, RequestVoteReply *reply) {
    pthread_mutex_lock(&c->ha.lock);
    uint64_t my_term = atomic_load(&c->ha.term);

    if (req->term < my_term) {        // stale candidate — reject
        reply->term = my_term;
        reply->vote_granted = 0;
        goto done;
    }
    if (req->term > my_term) {        // update term from candidate
        atomic_store(&c->ha.term, req->term);
        my_term = req->term;
        // Clear voted_for on new term transition
        c->ha.voted_for_term = 0;
        c->ha.voted_for_name[0] = '\0';
    }

    // Catch-up gate (Clarification 6): behind nodes must not influence elections
    if (c->ha.in_catch_up) {
        reply->vote_granted = 0;
        goto fill_reply;  // still send peer_config_tag/lsn so candidate has full picture
    }

    uint64_t my_tag = c->myconf->config_version_tag;
    uint64_t my_lsn = atomic_load(&c->persist.applied_lsn);
    int log_ok = (req->config_tag > my_tag) ||
                 (req->config_tag == my_tag && req->last_log_lsn >= my_lsn);

    // One vote per term: grant only if not yet voted, or voted for same candidate
    int already_voted = (c->ha.voted_for_term == my_term &&
                         strcmp(c->ha.voted_for_name, req->candidate_id) != 0);
    reply->vote_granted = (log_ok && !already_voted) ? 1 : 0;

    if (reply->vote_granted) {
        c->ha.voted_for_term = my_term;
        strncpy(c->ha.voted_for_name, req->candidate_id, XCE_MAXNAME_LEN - 1);
        ha_persist_term(c, my_term, req->candidate_id);  // durable before reply
    }
done:
    reply->term             = my_term;
    reply->peer_config_tag  = my_tag;
    reply->peer_last_log_lsn = my_lsn;
    pthread_mutex_unlock(&c->ha.lock);
}
```

**Election safety extensions**:

1. **Term + `voted_for` persistence** (`ha-term.bin` in `{workdir}/`):
   New `ha_persist_term(cluster_t *c, uint64_t term, const char *voted_for)` writes
   a 4-field struct atomically (write + rename) before any vote is granted or term
   incremented. On startup, `ha_init()` reads `ha-term.bin` and restores `ha.term`,
   `ha.voted_for_term`, `ha.voted_for_name`. Without this, a restarting node resets
   to term=0 and can double-vote in a term it already participated in.

2. **New fields in `ha_state_t`** (`ha.h`):
   ```c
   uint64_t voted_for_term;                    // term of last granted vote
   char     voted_for_name[XCE_MAXNAME_LEN];   // candidate name of last granted vote
   int      primary_ctrl_fd;                   // persistent control channel fd (Change F; -1 = none)
   bool     defer_config_fetch;                // Change C: set when tag=0 and peers have config;
                                               // cleared after post-election fetch completes
   ```

3. **Retire `ha_check_priority_stepdown()`** (ha.c:322–368):
   This function's guard `if (my_term > 0 || sender_term > 0) return` makes it
   permanently unreachable after the first quorum election (term is always ≥ 1
   thereafter). Change B explicitly removes the call site from the heartbeat handler
   and deletes the function body. Term-based leadership is fully handled by the
   `reply.term > my_term` check in `handle_request_vote()` and the existing
   `ha_do_demote()` path.

**Key properties**:
- **Most up-to-date wins**: A candidate that won quorum still yields if any *responding*
  peer has higher config_tag or (same tag, higher LSN). Non-responding peers never
  block election — only reachable peers with fresher state can cause a yield.
- **Config freshness hard gate**: Yielding due to fresher peer returns 0 (standby, not
  fatal). Only `config_tag == 0` with peers having `config_tag > 0` is fatal (-ENODATA),
  because that means the node has never been configured at all.
- **Split vote prevention (primary)**: Each node waits `election_timeout + rand_jitter`
  before candidacy (Change A `ha_wait_for_heartbeat_or_timeout()`). Different jitter
  values stagger candidacy attempts so the first node typically wins unopposed — the
  same xorshift64 `jitter_seed` used elsewhere. This is Raft's fundamental split-vote
  prevention, not just recovery.
- **Split vote recovery (safety net)**: If a split vote still occurs inside
  `ha_elect_evaluate_and_signal()`, a second attempt is scheduled with `rand_jitter`
  delay and incremented term. If that also fails, node returns 0 (standby). This
  safety net handles the rare case where multiple nodes have identical jitter values.
- **No priority tiebreak**: Priority is not used in Raft quorum election. When
  config_tag and LSN are identical across all peers, the jitter-based pre-election
  timeout is the tiebreaker. The `ha.my_priority` field is preserved in `ha_state_t`
  for backward compatibility but plays no role in election outcome.

### Change C — Pre-election config bootstrap (deferred fetch)

A node with `config_tag=0` is a config-seeker, not a candidate. Before calling
`am_i_primary_quorum()`, `bootstrap_config_if_needed()` determines whether config
exists among peers, but **defers the actual fetch until the primary is known** —
avoiding a speculative download from a peer that may not win the election.

**Rationale**: Fetching from the highest-tag *reachable* peer before election can
waste I/O: a fresher peer may start late, win the election, and push its config
anyway. Since the election already proves who has the freshest state, bootstrap
should defer to that proof rather than guessing.

New function `bootstrap_config_if_needed()` in config_repl.c (called from myapp.c):
```c
void bootstrap_config_if_needed(cluster_t *c) {
    if (c->myconf->config_version_tag != 0) return;  // already have config

    int retry_ms = 500;
    for (;;) {
        uint64_t best_tag = 0;
        bool any_reachable = false;

        for (int i = 0; i < c->pcfg.n_candidates; i++) {
            if (strcmp(c->pcfg.candidates[i].name, c->myconf->name) == 0) continue;
            uint64_t peer_tag = 0;
            int rc = xce_probe_config_tag(c->pcfg.candidates[i].host,
                                          c->pcfg.candidates[i].port, &peer_tag);
            if (rc != 0) continue;   // peer unreachable — skip
            any_reachable = true;
            if (peer_tag > best_tag) best_tag = peer_tag;
        }

        if (!any_reachable) {
            WARN("No config and no peer reachable — retrying in %dms (kill to abort)",
                 retry_ms);
            usleep((useconds_t)retry_ms * 1000);
            retry_ms = MIN(retry_ms * 2, 10000);
            continue;
        }

        if (best_tag == 0) {
            INFO("All peers also have no config — first-boot cluster, proceeding");
            return;   // all-zero cluster: first boot, proceed to election as-is
        }

        // Config exists among peers — defer fetch until primary is known.
        // The election will prove who has the freshest state.
        INFO("Config exists among peers (best tag=%016" PRIx64 ") — "
             "deferring fetch until primary is known", best_tag);
        c->ha.defer_config_fetch = true;
        return;
    }
}
```

The function never returns an error. If no peer is reachable, it retries with
exponential backoff. If all peers are `tag=0`, it returns for first-boot election.
If config exists among peers, it sets `defer_config_fetch` and returns — the node
will fetch from the proven winner after the election resolves.

`xce_probe_config_tag()` is a lightweight new RPC (single round-trip) that returns
the peer's current `config_version_tag` without transferring the full config payload.
Reuses the existing SSL connection setup pattern (`connect_ssl_raw()`).

**Cap'n Proto schema for `OP_probeConfigTag`**:
```c
struct ProbeConfigTagRequest {
  name @0 :Text;   // requester's node name (for logging on peer side)
}
struct ProbeConfigTagReply {
  config_tag @0 :UInt64;  // peer's current config_version_tag (0 if no config)
}
```
Answered by all nodes in any FSM state (including `CTRL_INIT`) — the dispatch gate
explicitly allows `OP_probeConfigTag` through before the controller is ready for
normal requests (Change A). Note: the actual config transfer (`OP_replConfig`) does
NOT need to be role-agnostic — with deferred fetch, `do_one_config_fetch()` is called
post-election when the serving peer is already in `CTRL_STANDBY` or `CTRL_PRIMARY`
and full dispatch is active. Only the probe op needs the `CTRL_INIT` exemption.

Integrated into `myapp.c` (before election):
```c
bootstrap_config_if_needed(&c);  // returns immediately: tag>0, first-boot, or defer

// The tag=0 node participates in the election as a voter but never wins
// against nodes with tag>0 (log_ok gate ensures this). It votes, discovers
// the winner via voted_for, and fetches from the proven primary.
rc = am_i_primary_quorum(&c);
if (rc < 0) return rc;

if (c.ha.defer_config_fetch && rc != 1) {
    // Fetch from the proven winner (known via voted_for or heartbeat source).
    // Guard rc != 1: a winning node has tag>0 so defer_config_fetch is never
    // set for it, but the explicit guard prevents double load_primary_config()
    // if logic shifts during implementation.
    do_one_config_fetch(c->ha.primary_host, c->ha.primary_port, 0);
    load_primary_config(&c);
    c->ha.defer_config_fetch = false;
}

if (rc == 1) {
    rc = load_primary_config(&c);  // load into memory (already on disk)
    if (rc != 0) return rc;
    atomic_store(&c->ha.role, CTRL_PRIMARY);
}
```

### Change D — Startup pre-flight checks + ASAN leak fix

`main.c`, `src/controller/cluster.c`, `lib/netio.c`, `ha.c`

1. **`--vcpus N` CLI parameter** (`main.c`, `src/controller/cluster.c`):
   Add a `--vcpus N` flag parsed in `main.c` alongside the existing `--peers` and `--name`
   flags. Default: `8` when not specified. Values `< 1` are rejected at startup with
   `"ERROR: --vcpus must be ≥ 1"` and exit `-EINVAL`. Remove the
   `vcpus` field from `cluster_config_t` (currently read from the config file). Pass the
   parsed value directly to `vtaskmgr_init()` — this removes `vtaskmgr_init()`'s
   dependency on `cluster_config`, enabling it to run before election (Change A).

   **Removed from cluster config schema**: the `vcpus:` key in the config file is no
   longer read. Existing configs that include it receive a logged deprecation warning
   (`"WARN: 'vcpus' in config file is deprecated — use --vcpus N instead"`) but the
   controller does not fail.

   **Rationale**: CPU count is a per-host capacity decision, not a cluster-wide policy.
   Different controller nodes in the same cluster may have different hardware. Keeping
   vcpus in the shared config forces all nodes to use the same value and couples an
   infrastructure tunable to the cluster state config file.

2. **Peer count validation in `main.c`** (before `cluster_init_phase1()`):
   The `--peers` list must contain exactly 1 node (single-node) or an **odd number ≥ 3**
   (HA mode). Even numbers cannot form a strict majority and are rejected at startup:
   ```c
   int n = pcfg.n_candidates;
   if (n != 1 && n % 2 == 0) {
       fprintf(stderr,
           "ERROR: --peers must list an odd number of controllers (1, 3, 5, ...).\n"
           "       %d controllers cannot form a majority. Use 1 for single-node\n"
           "       or add/remove one controller to reach an odd count.\n", n);
       return -EINVAL;
   }
   ```
   This removes the need for the `n_peers == 2` special case inside
   `am_i_primary_quorum()` — the quorum formula `(N+1)/2` is correct for all
   validated (odd) values of N.

3. **Cert pre-flight in `main.c`** (before `myapp_run()`): explicit `access()` check on
   `{workdir}/certs/xce.key` and `{workdir}/certs/xce.crt`. On failure: print
   operator-friendly `"ERROR: TLS cert not found: <path>"` and exit `-ENOENT`.
   Controller never starts without valid certs — same hard-prerequisite model as config.

4. **Fix ASAN leak in `ha_fini()` (ha.c:451)**: add `SSL_CTX_free(ha->ssl_ctx)` call.
   This closes the ~5320-byte direct+indirect leak shown in the pc13v1 ASAN output.

### Change E — Majority-commit WAL replication

`src/controller/wal.c` (or `persist.c`), `src/controller/ha.c`, `src/controller/cluster.c`

**Principle**: Primary does not ACK a write to its caller until ≥(N+1)/2 standbys have
confirmed they wrote the WAL entry to disk. This guarantees any promoted standby has
all committed entries — zero data loss on failover.

**New wire message: OP_walAck** (Cap'n Proto):
```c
struct WalAckRequest {
  term     @0 :UInt64;   // primary's current term (for stale-detection)
  lsn      @1 :UInt64;   // WAL entry LSN being ACKed
}
struct WalAckReply {
  term     @0 :UInt64;   // standby's term (primary steps down if reply.term > mine)
  lsn      @1 :UInt64;   // highest LSN standby has durably written
  success  @2 :Bool;
}
```

**Primary-side commit protocol** (`persist.c` / `wal.c`):
1. Write WAL entry to local disk (existing)
2. Fan-out `OP_walAck` to all connected standbys in parallel (async send)
3. Wait for ≥(N+1)/2 − 1 standby ACKs (own write counts as 1) within
   `commit_timeout_ms` (configurable, default 5000ms)
4. On quorum ACK: mark entry committed, return success to caller
5. On timeout without quorum: return `-ETIMEDOUT` to caller; entry is
   written locally but not committed — next primary will re-evaluate on
   WAL replay

**Standby-side WAL write** (existing replication path extended):
- On receiving replicated WAL entry: write to disk, fsync, send `OP_walAck` reply
- No change to standby application logic

**Backpressure**: Primary maintains a per-standby `pending_ack_count`. If a standby
falls more than `max_pending_acks` (default 64) entries behind, it is removed from the
active replication set and reconnects via existing catch-up path.

**Single-node mode**: quorum = 1, commit is immediate (own write = majority). No change.

**Clarification 1 — Durability boundary (which entry types block on majority-commit)**:

Not all WAL entry types have equal durability requirements. The rule:

| Entry type | Commit model | Rationale |
|---|---|---|
| `WAL_JOB_SUBMIT` | Majority-commit (blocking) | Job loss on failover is visible to user |
| `WAL_JOB_START` / `WAL_JOB_FINISH` | Majority-commit (blocking) | State transitions must survive failover |
| `WAL_QUEUE_CHANGE` | Majority-commit (blocking) | Queue config changes must be durable |
| `WAL_HOST_STATE` | Best-effort async | Reconstructed from worker reconnect on failover |
| `WAL_CHECKPOINT_REF` | Best-effort async | Checkpoint is self-contained; ref is advisory |

The existing `persist_wal_sync_append()` path (used for `WAL_JOB_SUBMIT` today with
ANY-standby wait) is extended to use majority quorum. The existing `persist_wal_async()`
path remains fire-and-forget for best-effort types. No new caller API — entry type
determines which path is taken inside `persist.c`.

**Clarification 2 — `committed_lsn` data model**:

Two new fields added to `persist_t` (persist.h):
```c
_Atomic uint64_t committed_lsn;  // highest LSN majority-ACKed; safe to serve/trim
uint64_t         applied_lsn;    // existing: highest LSN written locally (may exceed committed)
```

`committed_lsn` is advanced by `repl_wait_majority_ack()` after quorum is reached.
On promotion, the new primary replays WAL from its checkpoint up to `committed_lsn`
only — entries above `committed_lsn` are discarded (never committed to majority, so
no client received success for them). WAL trim guard updated: trim threshold = `committed_lsn`,
not `applied_lsn`.

Catch-up standbys (in_catch_up == 1) are excluded from quorum counting — they are not
yet in the active replication set. This means a cluster with 2-of-3 nodes catching up
cannot make progress until at least one catch-up completes. This is the correct CP
behavior: prefer unavailability over serving uncommitted state.

**Clarification 3 — `repl_broadcast()` prerequisite**:

`repl_broadcast()` (repl.c:183–196) is currently a stub. Majority-commit requires a
real broadcast implementation. This is co-developed with Change E — they are the same
work unit. Change E scope explicitly includes:
- Implement `repl_broadcast()` with per-standby TCP send via existing iomgr pipeline
- Implement `repl_wait_majority_ack()` replacing the existing `repl_wait_ack()` (any→majority)
- Add per-standby send error callback: failed send → decrement active standby count,
  trigger catch-up reconnect (existing path)

If `repl_broadcast()` stub lands separately before Change E, Change E still works —
it replaces the stub. No ordering dependency beyond Task 11 (config fetch).

**New function `persist_truncate_wal_above(persist_t *p, uint64_t cutoff_lsn)`** (`persist.c`):

Removes all WAL entries with LSN > `cutoff_lsn` from local storage. Called in two
contexts (Clarifications 4 and 5). Implementation: scan WAL segment files from newest
to oldest; truncate the file containing `cutoff_lsn` at the byte offset immediately
after that entry; delete any segment files that start above `cutoff_lsn`. Updates
`p->applied_lsn = cutoff_lsn` atomically after truncation. Must be idempotent (calling
twice with same cutoff is safe). New function — does not currently exist in `persist.c`.

**Clarification 4 — `committed_lsn` must be piggybacked on heartbeats**:

A promoted standby needs to know where to truncate its WAL. Without this it only has
`applied_lsn`, which may include uncommitted orphan entries written by the old primary
but never majority-ACKed.

- Extend `CtrlHeartbeat` Cap'n Proto struct with `committed_lsn @N :UInt64`
- Standbys store it in `ha.last_known_committed_lsn` (`_Atomic uint64_t`, init 0) on
  each heartbeat receipt
- On promotion, immediately after `am_i_primary_quorum()` returns 1:
  ```c
  persist_truncate_wal_above(&c->persist, c->ha.last_known_committed_lsn);
  c->persist.committed_lsn = c->ha.last_known_committed_lsn;
  ```
- First-boot failover (no heartbeat ever received): `last_known_committed_lsn = 0`,
  so WAL replay starts from checkpoint only — correct and safe

**Clarification 5 — Log divergence on standby re-join**:

The existing catch-up path assumes the standby is simply *behind* the primary (a clean
subset). After a primary crash, a standby may have orphaned WAL entries above the new
primary's `committed_lsn` — entries the old primary replicated but never majority-ACKed.
Replaying those entries would corrupt the standby's state.

Catch-up handshake extension (Change E scope):
1. Primary sends its `committed_lsn` in the catch-up initiation reply
2. Standby calls `persist_truncate_wal_above(&c->persist, primary_committed_lsn)` before
   requesting WAL replay — discards any orphaned entries above that boundary
3. Standby sets `applied_lsn = primary_committed_lsn` as the new replay base
4. Normal WAL catch-up (existing mechanism) proceeds from there

**Clarification 6 — Catch-up standbys excluded from voting (not just quorum count)**:

Clarification 2 excludes `in_catch_up==1` standbys from the majority quorum count.
They must also be prevented from *granting* votes — a behind-node's stale LSN could
tip an election to a less-fresh candidate.

In `handle_request_vote()`, before the `log_ok` freshness check:
```c
if (c->ha.in_catch_up) {
    reply->vote_granted = 0;
    // still populate peer_config_tag/peer_last_log_lsn — candidate needs the full picture
    goto fill_reply;
}
```

**Clarification 7 — `applied_lsn` in vote request must be post-fsync**:

`RequestVoteRequest.last_log_lsn` is sourced from `atomic_load(&c->persist.applied_lsn)`.
This is correct only if `applied_lsn` is advanced *after* `fsync()` completes in
`persist.c`. A node that claims an LSN it has not durably written could win an election
and then fail to replay entries it advertised. Verify in `persist.c` before implementing
Change B — if `applied_lsn` is advanced pre-fsync anywhere, fix that first.

### Change F — Persistent standby↔primary control channel

**Files**: `src/controller/ha.c`, `src/controller/myapp.c`, `src/controller/cluster.c`,
`src/controller/repl.c`

**Background**: Standbys currently detect primary failure through `ha_check_heartbeat_nocluster()`
(ha.c:380–405), which polls a `last_heartbeat` timestamp with a 10s floor. Failover detection
latency is up to 10 seconds. Additionally, each config fetch and WAL replication message opens a
new TCP+SSL connection (`connect_ssl_raw()`, config_repl.c:685–775), adding per-message handshake
latency for WAL streaming (Change E).

**Changes**:

1. **Standby discovers and connects to primary** — new `ha_find_and_connect_to_primary(cluster_t *c)`
   (called in `myapp.c` standby path after election). Opens a TCP+SSL connection to the primary
   peer and registers the fd with epoll `EPOLLIN | EPOLLRDHUP`. The connection is stored in
   `ha_state_t` as `ha->primary_ctrl_fd`. On connect, the standby sends an extended `OP_ctrlHello`
   (item 5) carrying identity and state. The primary registers the standby, initiates catch-up,
   and begins streaming any missing WAL before the standby enters the active replication set.

   **Primary discovery algorithm** (how the standby finds the winner's address):

   Two startup paths lead here with different amounts of prior knowledge:

   - **Heartbeat path** (`CTRL_INIT → CTRL_STANDBY`): `ha_wait_for_heartbeat_or_timeout()` received
     a heartbeat/`OP_ctrlHello` from an existing primary. The sender's host/port is in the packet.
     Store immediately as `ha.primary_host/port`. No further discovery needed — connect directly.

   - **Election path** (`CTRL_ELECTING → CTRL_STANDBY`): lost `am_i_primary_quorum()`. Discovery
     uses a three-tier strategy exploiting Raft's structural property that the majority voted for
     the winner:

   ```
   Tier 1 (fast path — O(1) for ≥(N+1)/2 nodes):
     If ha.voted_for_name is set:
       try ha_connect_to_primary(voted_for peer)
       on success: store ha.primary_host/port, done
       on XCE_ERR_NOT_READY: peer still electing — backoff, retry tier 1

   Tier 2 (redirect — O(1) once any majority member connected):
     Probe remaining peers (random order) with OP_ctrlHello
     A CTRL_STANDBY peer that already knows the primary responds with
       XCE_ERR_STANDBY + redirect_host/port (same field as Change G)
     Follow redirect immediately — connect to that address

   Tier 3 (blind scan fallback — for split-vote minorities):
     If redirect not received: try remaining peers sequentially
     A CTRL_PRIMARY peer accepts OP_ctrlHello normally → connected
     A CTRL_INIT/CTRL_ELECTING peer returns XCE_ERR_NOT_READY → skip+retry
     Repeat with 100ms → 5s exponential backoff until primary found
   ```

   **Why this converges quickly**: In a 3-node cluster, at least 2 nodes voted for the winner.
   Those 2 connect via Tier 1 immediately. The 3rd probes either of them (Tier 2), gets a
   redirect, and connects. Total: 2 direct + 1 redirect = all standbys connected with at most
   one extra probe per minority node. In a 5-node cluster, at least 3 connect via Tier 1; the
   remaining 2 get redirected.

   **`ha.primary_host/port` update**: set on successful connect; cleared to empty on
   `EPOLLRDHUP` (primary disconnect). During the election window it is empty; standbys receiving
   `OP_ctrlHello` while `ha.primary_host` is empty reply with `XCE_ERR_STANDBY` and no redirect
   (graceful — probing node falls through to Tier 3).

   **No new message types**: the redirect uses the existing `XCE_ERR_STANDBY` error reply fields
   already extended in Change G (`redirect_host`, `redirect_port`). The standby dispatch gate
   (already returning `XCE_ERR_STANDBY` for primary-only ops) populates these fields from
   `ha.primary_host/port` when they are known.

2. **Standby join synchronization — full state sync on connect** (cluster.c, ha.c, repl.c):

   A standby with stale WAL cannot safely promote — its in-memory state (jobs, queues,
   resources) would be out of date, and promoted writes would replay entries clients already
   saw as committed. Every standby must be fully current before it enters the quorum.

   **Extended `OP_ctrlHello` Cap'n Proto schema**:
   ```c
   struct CtrlHelloRequest {
     name        @0 :Text;    // peer name (from --peers)
     key         @1 :Data;    // cluster auth key: SHA-256(cluster_name + shared_secret)
                              // shared_secret read from myconf->cluster_key (new config field)
                              // primary rejects connections where HMAC does not match its own
     applied_lsn @2 :UInt64; // standby's current applied_lsn (0 if fresh node)
     config_tag  @3 :UInt64; // standby's current config_version_tag (0 if none)
   }
   struct CtrlHelloReply {
     committed_lsn  @0 :UInt64; // primary's current committed_lsn
     needs_snapshot @1 :Bool;   // true if standby's WAL predates oldest primary segment
     config_tag     @2 :UInt64; // primary's current config_version_tag
   }
   ```

   **New wire messages for sync**:
   ```c
   struct SnapshotTransfer {
     snapshot_lsn @0 :UInt64; // LSN of this snapshot (= checkpoint_lsn after fresh checkpoint)
     payload      @1 :Data;   // serialized snapshot blob (from persist_checkpoint())
   }
   struct WalCatchupEntry {
     lsn     @0 :UInt64;      // WAL entry LSN
     type    @1 :UInt8;       // WAL entry type (WAL_JOB_SUBMIT etc.)
     payload @2 :Data;        // WAL entry payload
   }
   ```
   `OP_snapshotTransfer` and `OP_walCatchupEntry` flow on the persistent control channel fd —
   no new connections.

   **Primary-side `handle_ctrlhello()` sequence** (new handler in cluster.c):
   ```
   1. Extract name, applied_lsn, config_tag from CtrlHelloRequest
   2. repl_add_standby(&c->repl, ep, host, port)
   3. needs_snap = repl_needs_snapshot(&c->repl, applied_lsn, oldest_wal_lsn())
   4. Reply: CtrlHelloReply{c->persist.committed_lsn, needs_snap, myconf->config_version_tag}
   5. repl_start_catchup(&c->repl, ep, applied_lsn)   // sets standby's in_catch_up=1
   6. If needs_snapshot:
        persist_checkpoint(&c->persist)               // flush in-memory state to disk
        send OP_snapshotTransfer(snapshot_lsn, blob)
        applied_lsn = snapshot_lsn                    // advance standby's effective base
   7. Streaming loop — runs until standby is fully current:
        target_lsn = atomic_load(&c->persist.committed_lsn)   // read current live value
        stream WAL entries [applied_lsn+1 → target_lsn] via OP_walCatchupEntry
        wait for OP_walAck with acked_lsn >= target_lsn
        if atomic_load(&c->persist.committed_lsn) > target_lsn:
            applied_lsn = target_lsn; continue loop  // new entries committed, catch up again
        // standby is now at current committed_lsn — exit loop
   8. repl_complete_catchup(&c->repl, ep)   // in_catch_up=0; standby enters live replication
   9. If standby config_tag != primary config_tag:
        push_config_to_standbys(c, blob, blen, tag)
   ```

   **`repl_broadcast()` skips `in_catch_up==1` standbys**: while the streaming loop (step 7)
   is in progress, `repl_broadcast()` does not send to this standby. The loop itself handles
   delivery of all committed entries — including entries committed after the handshake — by
   re-reading `committed_lsn` on each iteration. The loop exits only when the standby's
   `acked_lsn` equals the *current* `committed_lsn`, at which point `repl_complete_catchup()`
   fires and live broadcast takes over. There is no window where entries can be missed.

   **Standby-side sequence** (after receiving CtrlHelloReply in ha.c / myapp.c):
   ```
   1. Store primary_committed_lsn = reply.committed_lsn
   2. persist_truncate_wal_above(&c->persist, primary_committed_lsn)  // Clarification 5
   3. ha.last_known_committed_lsn = primary_committed_lsn             // Clarification 4
   4. ha.in_catch_up = 1                                              // deny votes during sync
   5. If reply.needs_snapshot:
        wait for OP_snapshotTransfer
        write received blob as local snapshot file
        load_snapshot(&c->persist, path, snapshot_lsn)
        applied_lsn = snapshot.snapshot_lsn
   6. Receive OP_walCatchupEntry stream; call apply_wal_entry() for each
   7. Send OP_walAck after each entry (or configurable batch):
        OP_walAck{acked_lsn = last applied LSN}
   8. When primary sends no more OP_walCatchupEntry (catch-up loop ended on primary side):
        ha.in_catch_up = 0   // standby now in live replication; votes granted normally
   ```

   `apply_wal_entry()` is the same WAL replay function used during startup recovery — it
   applies a single WAL entry to the in-memory state (jobs, queues, resources). Must exist
   or be extracted from the startup replay path before this change lands.

   **`repl_needs_snapshot()` already exists** in repl.c — compares `standby_applied_lsn`
   against `oldest_wal_lsn` (the trim floor). If standby is behind the trim floor,
   `persist_checkpoint()` is called on the primary to produce a fresh snapshot, which is
   then streamed in full. This is the only path where `persist_checkpoint()` is called
   on-demand rather than on schedule.

   **`push_config_to_standbys()` is declared** in config_repl.h; config sync at join
   handles the case where a standby was offline during a config upgrade.

2. **Heartbeat runs on control channel** — primary sends periodic heartbeats on the controller↔controller
   channel (in addition to existing worker heartbeats). Standby updates `last_heartbeat` on receipt,
   but the epoll path is the fast lane.

3. **Replace timestamp polling with dual-trigger failover detection** —
   `ha_check_heartbeat_nocluster()` (ha.c:380–405) is retired. Two complementary paths:
   - **Fast path (`EPOLLRDHUP`)**: TCP disconnect detected immediately; `primary_ctrl_fd`
     closed and nulled before `am_i_primary_quorum()` is called. Sub-second latency.
   - **Fallback path (heartbeat timeout vtask)**: armed after each heartbeat is received
     (`vtaskmgr_schedule(election_timeout + rand_jitter, ha_heartbeat_timeout_cb)`); reset
     on every subsequent heartbeat. Fires when primary is silent but TCP is alive — covers
     network partitions that keep TCP open, primary overload, or missed heartbeat bursts.
     On fire: close+null `primary_ctrl_fd`, set `CTRL_ELECTING`, call `am_i_primary_quorum()`.

4. **WAL streaming over persistent channel (Change E complement)** — `repl_broadcast()`
   (repl.c:183–196) sends WAL entries on the persistent control channels established by each
   standby rather than opening per-entry connections. No new connection setup per WAL write.

**Role flip on promotion**: When a standby wins election and becomes primary, it must
close and discard its `primary_ctrl_fd` (the old connection to the now-dead primary)
*before* calling `am_i_primary_quorum()` triggers the role transition. After promotion,
the new primary accepts `OP_ctrlHello` connections from the remaining standbys — the
role of `primary_ctrl_fd` inverts from "my connection out" to "their connections in",
tracked in a per-standby fd table rather than a single field.

**Reconnect behavior**: On primary_ctrl_fd disconnect, standby closes and nulls
`primary_ctrl_fd` first, then calls `am_i_primary_quorum()`.
If it becomes primary, the old primary_ctrl_fd is discarded. If it remains standby, it reconnects
with exponential backoff (100ms, 200ms, 400ms, max 5s) until the new primary is reachable.

**Single-node mode**: `ha_connect_to_primary()` is a no-op when `n_peers == 1`.

**Key properties**:
- Sub-second failover detection (`EPOLLRDHUP` vs 10s timeout)
- No per-entry TCP+SSL handshake overhead for WAL replication
- Single liveness signal: connection alive = peer alive
- Backward-compatible: standby reconnect on disconnect triggers election (same semantics as today,
  faster latency)

### Change G — libxce client-side retry for transient controller errors

**Files**: `src/lib/` (libxce — client request/connection layer)

**Background**: With `CTRL_ELECTING` gating requests during the election window
(Change A), and standbys already returning `XCE_ERR_STANDBY` to misdirected
requests, every daemon (worker, scheduler) needs to handle these transient errors
gracefully. Without built-in retry in libxce, each daemon would need its own
ad-hoc loop — fragile and inconsistent.

**Changes**:

1. **Retry policy in libxce's client request path** (the function wrapping
   `xce_encode()` + send + recv + `xce_decode()`):
   - On `XCE_ERR_NOT_READY`: back off and retry against same endpoint
   - On `XCE_ERR_STANDBY`: **redirect** — the standby includes the current primary's
     `host:port` in the error payload when known (`ha.primary_host/port`). libxce
     reconnects directly to the redirected endpoint instead of blind backoff.
     When primary is unknown (e.g. during startup), falls back to normal retry.
   - On TCP-level `ECONNREFUSED` or `ETIMEDOUT`: reconnect and retry
     (handles the window before `cluster_open_listener()` runs on a starting peer)
   - Non-retriable errors (`XCE_ERR_AUTH`, `XCE_ERR_INVALID`, `XCE_ERR_NOENT`,
     `XCE_ERR_PERM`, etc.): surface immediately to caller

   **`XCE_ERR_STANDBY` wire extension**: add optional `redirect_host` (Text) and
   `redirect_port` (UInt16) fields to the error reply Cap'n Proto struct. Standbys
   populate these when `ha.primary_host` is non-empty; clients that don't understand
   the new fields simply retry with backoff (backward-compatible).

2. **Retry configuration** (sensible defaults, overridable per client):
   ```c
   typedef struct {
       int     max_retries;          // default 20
       int     initial_retry_ms;     // default 50
       int     max_retry_ms;         // default 5000
       double  backoff_multiplier;   // default 2.0 (exponential)
   } xce_retry_config_t;
   ```
   Defaults cover a full election round (up to 20 × 50ms–5000ms) without
   requiring callers to configure anything. Daemons that need tighter or looser
   bounds pass a custom `xce_retry_config_t`.

3. **Transparent to callers**: existing daemon code makes one `xce_request()` call;
   libxce handles retry internally. No changes needed in worker, scheduler, or
   apiserver code.

**Key properties**:
- Decouples daemon startup order — a worker that starts before the primary is ready
  simply waits for the retry window to expire rather than crashing or looping manually
- `XCE_ERR_STANDBY` retry enables transparent primary failover: after failover, the
  next request to the old primary (now standby) gets `XCE_ERR_STANDBY`, libxce retries
  against the same host/port and the new primary responds
- Backoff prevents thundering-herd when 50 workers restart simultaneously after a failover

### Change H — Majority-commit config push

`src/controller/config_repl.c`, `src/controller/repl.c`

**Principle**: Config upgrades (`OP_upgradeConfig`) use the same majority-commit
infrastructure as WAL replication (Change E). The primary does not ACK a config
change to the operator until ≥(N+1)/2 standbys have confirmed durable write.

**Why this matters**: Without majority-commit, a primary could accept a config change,
ACK to the operator, then crash. The new primary serves the old config, and the
operator's change is silently lost. For strict CP, config writes must have the same
durability guarantee as WAL writes — even though they are orders of magnitude rarer.

**Changes**:

1. **Reuse Change E infrastructure** — `repl_broadcast()` (sends to all connected
   standbys on persistent channels) and `repl_wait_majority_ack()` (blocks until
   ≥(N+1)/2 ACKs or `commit_timeout_ms`) already built for WAL. Both functions
   accept a message-type parameter so they can serve WAL (`OP_walAck`) and config
   (`OP_upgradeConfigAck`) ACKs without duplication. Zero new infrastructure.

2. **`push_config_to_standbys()` extended** (config_repl.c):
   ```c
   int push_config_to_standbys(cluster_t *c, const char *blob, size_t len, uint64_t tag) {
       // 1. Write locally (existing atomic write+rename — unchanged)
       int rc = config_repl_write_local(c, blob, len, tag);
       if (rc != 0) return rc;

       // 2. Fan-out OP_upgradeConfig via repl_broadcast (same channel as WAL)
       rc = repl_broadcast(c, OP_upgradeConfig, blob, len);
       if (rc != 0) return rc;

       // 3. Wait for ≥(N+1)/2 standby ACKs (own write = 1 vote)
       rc = repl_wait_majority_ack(c, c->ha.config.commit_timeout_ms);

       // 4. On quorum: advance committed_config_tag and return success
       if (rc == 0) {
           atomic_store(&c->myconf->committed_config_tag, tag);
       }
       return rc;  // -ETIMEDOUT on failure — operator retries
   }
   ```

3. **Standby-side config ACK** — on receiving `OP_upgradeConfig`, standby:
   - Writes config blob to disk (existing atomic write path)
   - Loads config into memory
   - Sends `OP_upgradeConfigAck` with `acked_tag = written_tag`
   - ACK semantics identical to `OP_walAck` — durable write complete

4. **`OP_upgradeConfigAck` Cap'n Proto**:
   ```c
   struct UpgradeConfigAck {
     term @0 :UInt64;      // standby's term (primary steps down if ack.term > mine)
     acked_tag @1 :UInt64; // config tag standby has durably written
     success @2 :Bool;
   }
   ```

5. **`committed_config_tag` field** added to cluster config state (analogous to
   `committed_lsn` for WAL). Advanced only on majority ACK. Stored alongside
   `config_version_tag` in durable state. On promotion, a new primary serves only
   configs up to `committed_config_tag`. **Piggybacked on heartbeats** (same as
   `committed_lsn` — Clarification 4): standbys store the latest value in
   `ha.last_known_committed_config_tag` so a promoted standby knows the safe
   boundary without an extra round-trip. First-boot promotion (no heartbeat
   received): value is 0, standby serves whatever config it loaded from disk —
   correct for first-boot where no upgrade has been committed.

**Key properties**:
- Same `commit_timeout_ms` as WAL (configurable, default 5000ms)
- Single-node mode: quorum=1, commit is immediate (own write = majority)
- Config upgrades are operator-initiated and rare — the blocking ACK wait adds
  negligible overhead to cluster operations
- Catch-up standbys (`in_catch_up==1`) are excluded from quorum count,
  consistent with WAL majority rules
- If commit times out, operator gets `-ETIMEDOUT` and can retry — libxce retry
  (Change G) handles this transparently for the API server

## Scope

- **In**:
  - Change A: Extract `cluster_open_listener()`, `iomgr_init()`, and `vtaskmgr_init()`
    from phase2; call all three before election in myapp.c so the full connection
    framework is running during election (incoming `OP_requestVote` handled by normal
    dispatch switch); add `CTRL_ELECTING` role + pre-dispatch gate returning
    `XCE_ERR_NOT_READY` for non-vote requests during election window
  - Change B: Full Raft-style quorum voting in `am_i_primary_quorum()`
    - New `OP_requestVote` wire protocol (Cap'n Proto request + reply)
    - `handle_request_vote()` handler on all peers with `voted_for` enforcement
    - Config tag + LSN freshness comparison in vote grant and candidate yield gate
    - Hard gate A (fatal): refuse primary if config_tag=0 and any peer has tag>0
    - Hard gate B (yield): winner yields if any *responding* peer has fresher data
    - Split vote retry with jitter (one retry after timeout)
    - Term + `voted_for` persistence to `ha-term.bin` (survives restart)
    - New `ha_state_t` fields: `voted_for_term`, `voted_for_name`, `primary_ctrl_fd`
    - Retire `ha_check_priority_stepdown()` (permanently unreachable after first election)
  - Change C: `bootstrap_config_if_needed()` pre-election bootstrap — probes peers
    for config tags, defers fetch until primary is known (avoids speculative download
    from a peer that may not win the election); tag=0 node participates as voter only;
    new `xce_probe_config_tag()` lightweight RPC; role-agnostic
    `handle_config_request()` so peers can serve config to a bootstrapping node
  - Change D: `--vcpus N` CLI parameter + remove `vcpus` from cluster config (deprecation
    warning for existing configs); odd-N peer-count validator at startup (`-EINVAL` on
    even count); cert pre-flight check in `main.c`; `SSL_CTX_free` ASAN fix in `ha.c`
  - Change E: Majority-commit WAL replication — `OP_walAck` protocol, primary
    commit gating, standby ACK path, backpressure, `commit_timeout_ms` config
  - Change F: Persistent standby↔primary control channel — `ha_connect_to_primary()`,
    `EPOLLRDHUP`-based failover detection, heartbeat on control channel, WAL streaming
    over persistent connection
  - Change G: libxce client-side retry — exponential backoff on `XCE_ERR_NOT_READY`,
    `XCE_ERR_STANDBY`, `ECONNREFUSED`, `ETIMEDOUT`; transparent to daemon callers;
    configurable via `xce_retry_config_t` with safe defaults
  - Change H: Majority-commit config push — `OP_upgradeConfig` uses the same
    `repl_broadcast()` + `repl_wait_majority_ack()` infrastructure as WAL replication
    (Change E). Primary waits for ≥(N+1)/2 standby ACKs before confirming a config
    upgrade to the operator. Reuses `commit_timeout_ms`. Zero new infrastructure cost.
  - Unit and integration tests for all eight changes
- **Out**:
  - Raft snapshot protocol — existing dual-zarray COW is orthogonal and unchanged
  - Joint consensus for cluster membership changes — candidate list is static
  - do_one_config_fetch() stub wiring (Task 11) — prerequisite; must land first or co-developed
  - mTLS between controllers — trusted-LAN model with server-side TLS is sufficient

## Boundary Candidates

1. **Election protocol** — full replacement of `am_i_primary()` with quorum voting
2. **Config wire protocol** — new OP_requestVote, OP_probeConfigTag, OP_upgradeConfigAck; existing OP_replConfig used for post-election deferred fetch
3. **Primary startup path** — bootstrap integration in myapp.c
4. **Listener lifecycle** — extracted from phase2 to phase1

## Out of Boundary

- Standby fetch path (`fetch_config_from_primary()`) — untouched
- Config format, versioning, or storage layout — untouched
- Raft snapshot protocol — existing dual-zarray COW is orthogonal
- Joint consensus for cluster membership changes — candidate list is static
- mTLS — trusted-LAN model with server-side TLS is sufficient

## Upstream / Downstream

- **Upstream**: Task 11 (`do_one_config_fetch()` stub) on persistent_ha branch — with deferred fetch (Change C), the probe-only path in `bootstrap_config_if_needed()` does not call `do_one_config_fetch()`. Task 11 is hit only at the post-election deferred fetch step in `myapp.c`. If Task 11 is not yet landed, the deferred fetch returns `-ENOSYS` and the bootstrapping standby fails to load config — a fatal but clearly logged error. Both can be co-developed; Task 11 must land before or with Change C for a functional cluster.
- **Downstream**: Phase 3 hardening (membership changes, leader transfer), ops tooling for election/commit observability, monitoring metrics for WAL replication lag.

## Existing Spec Touchpoints

- **Adjacent**: none of the active specs (resval-id-decoupling, scheduler-controller-msg-gap, worker-join-refactor) touch config replication, HA election, or cluster init.

## Constraints

- C (gnu17), no C++, no new external dependencies.
- ASAN build must pass (`nix build .#xcecore_debug`).
- **`--peers` must list 1 or an odd number ≥ 3 controllers.** Even counts are rejected
  at startup with `-EINVAL`. This simplifies quorum math: `(N+1)/2` is exact for all
  valid N; no special-case branches needed inside `am_i_primary_quorum()`.
- **Static membership**: the candidate list is fixed at startup from `--peers` and never
  changes while the cluster is running. No dynamic join/leave of controller peers.
  Worker and scheduler nodes may still join dynamically — only controller peer membership
  is static. This eliminates the need for joint consensus (Raft §6) and locks the quorum
  denominator for the lifetime of the cluster.
- Single-node mode unaffected: quorum=1 everywhere (election, WAL commit = own write).
- TDD required: failing test first, then implementation.
- All new C test cases go in `src/controller/testsuite/` before implementation.
- `make test` must pass without a live server.
- `commit_timeout_ms` must be configurable (default 5000ms) to allow tuning per deployment latency.
- **WAL write stall during standby catch-up (operational constraint)**: In a 3-node cluster,
  quorum for WAL commit is 2 (primary + 1 standby). Between primary winning election and the
  first standby completing `repl_complete_catchup()`, the active replication set has 0 standbys.
  Every `persist_wal_sync_append()` call blocks for `commit_timeout_ms` and returns `-ETIMEDOUT`.
  This is CP-correct behavior — the cluster is unavailable until quorum is formed. Operators
  should expect a stall of up to `commit_timeout_ms` per write during the initial catch-up
  window on first boot or after a restart. Jobs submitted during this window receive a retriable
  error; libxce retry (Change G) handles this transparently for workers and schedulers.

## Deployment Reference

### Controller node topology

| Cluster size | Failure tolerance | WAL ACKs needed | Recommendation |
|---|---|---|---|
| 1 node | 0 | self only | dev/test only |
| 3 nodes | 1 failure | 1 standby ACK | minimum for production |
| 5 nodes | 2 failures | 2 standby ACKs | recommended for critical HPC |
| 7 nodes | 3 failures | 3 standby ACKs | rarely justified; adds latency |

### Job submission throughput (Raft overhead only)

Basis: LAN environment, 5 controller nodes, quorum = 3 (primary + 2 standby ACKs).

**Critical path per `WAL_JOB_SUBMIT`:**

```
client → primary:        RTT/2 + primary WAL fsync
primary → 2 standbys:   RTT/2 + standby WAL fsync  (parallel, overlaps with above)
2nd ACK → primary:      RTT/2
primary confirms:        RTT/2
─────────────────────────────────────────────────
Total:  1 LAN RTT + max(fsync_primary, fsync_standby)
```

| Parameter | Optimistic | Conservative |
|---|---|---|
| LAN RTT (controller↔controller) | 0.1 ms | 0.5 ms |
| NVMe fsync latency | 0.05 ms | 0.2 ms |
| Effective commit latency | ~0.2 ms | ~0.7 ms |

**Throughput estimates:**

| Submit model | Commit latency | Jobs/sec |
|---|---|---|
| Serial single stream | 0.3–1 ms | 1,000–3,300 |
| 10 concurrent clients | 0.5 ms | ~20,000 |
| 100 concurrent clients | 0.5 ms | ~200,000 (non-Raft bottlenecks dominate before this) |

**Practical HPC ceiling**: 5,000–20,000 jobs/sec before jobmgr queue locking and
in-memory state mutations become the constraint.

**5-node vs 3-node write latency**: negligible. Majority-commit waits for the
**2nd-fastest** ACK out of 4 standbys (5-node) vs the **1st** out of 2 (3-node).
Under uniform LAN latency the difference is sub-millisecond. The extra two nodes
cost nothing in write throughput but double the failure tolerance.

**When Raft becomes a bottleneck**: automated pipelines submitting >10,000 short
jobs/sec. Mitigation: batch multiple job submits into a single WAL entry at the
API layer before this design makes that necessary.