myxce/.kiro/specs/ha-config-bootstrap/brief.md

Brief: ha-election-raft-config

Problem

The current LSF-style priority-based election has three critical safety gaps:

1. Config freshness not guaranteed: A node with missing or stale config can be elected primary and serve incorrect cluster state to workers/schedulers.

2. False self-elections during concurrent startup: When pc13v1 starts while pc13v (the priority-0 primary) is still initializing, pc13v's peer listener port is not open yet. pc13v1's election probe gets ECONNREFUSED, falsely concludes the primary is absent, and self-elects. Both nodes declare themselves primary.

3. No bootstrap path for genuine failover: When a standby is promoted during real failover (original primary down), it has no mechanism to pull config from peer standbys if its local conf/current is missing.

Reproduction from logs:

• pc13v starts at 12:30:03, pc13v1 starts at 12:30:22 (19s later)
• pc13v1 probes pc13v:9990 during election, times out within 3046ms
• pc13v is healthy but still in load_primary_config() or early phase2 — listener not open
• pc13v1 wins: "no higher-priority candidate responded"
• pc13v1 calls load_primary_config() → "Primary has no config at '/tmp/xce/pc13v1/conf/current' (rc=-2)" → fatal exit

Current State

• Election (ha.c:243–306): LSF-style priority-based. Priority-0 always wins unconditionally; priority-N probes [0..N-1] and self-elects on timeout. No quorum. No config-version check. Logically correct for simple cases but unsafe for config freshness.
• Listener timing (myapp.c:21–96): bind()+listen() happens inside cluster_init_phase2() AFTER election and config load. Creates a 2–3 second window where the primary is unreachable to probes.
• Config versioning (config_repl.c): SHA-256 tagged, atomic writes, retention policy, staleness detection (conf_stale flag). Complete and correct.
• Config fetch (config_repl.c:891–965): standby→primary pull framework exists; do_one_config_fetch() returns -ENOSYS (stub, Task 11 on branch).
• Bootstrap after failover: not implemented. Design §14.4 deferred quorum consensus.
• Simultaneous primaries: ha_check_priority_stepdown() (ha.c:322) has a term=0 exact check that fails to resolve races when both nodes increment term before heartbeat exchange.

Availability Trade-off (CP chosen)

This design chooses Consistency over Availability (CP in CAP terms):

• A cluster of N nodes requires a majority (≥(N+1)/2) for both election and write commits.
• Election: A minority partition cannot elect a primary. The cluster is unavailable until the majority partition is restored.
• Writes (WAL replication): Primary waits for majority standby ACKs before confirming a write (job submit, queue change, state update). In-flight writes that cannot reach majority are rejected. This adds ~1 LAN RTT per write (~0.1–5ms depending on network).
• Zero data loss on failover: A promoted standby is guaranteed to have all committed WAL entries because majority replication ensures durability before write ACK.
• Operational implication: Plan for N≥3 nodes. A 2-node cluster has no meaningful majority; it is equivalent to single-node with a cold standby. For HPC, zero data loss (no lost job submissions, no stale queue state) outweighs the write-latency cost.

Desired Outcome

1. Raft-style quorum election (CP): A candidate must obtain ACKs from a strict majority of peers (≥(N+1)/2) before becoming primary. A minority partition cannot elect a primary. The cluster is unavailable until the majority partition is restored.

2. Config freshness encoded in votes: Each vote reply includes the peer's config tag and WAL LSN. The candidate with the freshest config (highest tag, then highest LSN) among quorum members wins tiebreaks.

3. No-config nodes bootstrap after election (deferred fetch): A node with config_tag=0 never initiates candidacy. It probes peers to determine whether config exists anywhere, but defers the actual fetch until the primary is known — the election proves who has the freshest state, so bootstrap fetches from the proven winner rather than speculatively guessing. Three sub-cases:

   • tag=0, peers have config → set defer_config_fetch, participate as voter only, fetch from winner after election
   • tag=0, all peers also tag=0 → first-boot cluster, proceed to election as-is
   • tag=0, no peer reachable → retry: probe loop with exponential backoff (500ms→10s); log WARN every attempt; block indefinitely until a peer responds or admin kills

4. Listener before election: Peer listener socket opens before election so concurrent-startup probes reach an open port. False self-elections during overlapping startup are eliminated.

5. Config freshness preserved through all paths: A node always has a valid config before entering phase 2 — either from local disk (normal case) or fetched from the proven election winner after the election resolves (no-config case, deferred fetch in Change C). A quorum winner therefore always has config_tag > 0 when peers do, and serves correct cluster state immediately after promotion with no additional bootstrap step.

6. Most up-to-date primary: Among all reachable nodes, only the one with the highest config_tag (then highest LSN) can become primary. A candidate that wins the vote count but sees a fresher responding peer yields back to standby — the fresher peer runs its own election and wins. Non-responding (partitioned or crashed) peers never block election.

7. No simultaneous primaries: Only one node can obtain a quorum at any given term. Term-based fencing prevents split-brain on partition heal. voted_for tracking (persisted to ha-term.bin) enforces one vote per term per node across restarts.

8. Majority-commit WAL replication (CP writes): Primary waits for majority (≥(N+1)/2) standby ACKs before marking a WAL entry committed and confirming the write to the caller. In-flight writes that cannot reach majority within a timeout are rejected. Zero data loss on failover: promoted standby has all committed entries.

9. Majority-commit config push (CP config): Config upgrades (OP_upgradeConfig) use the same repl_broadcast() + repl_wait_majority_ack() infrastructure as WAL replication — the primary waits for ≥(N+1)/2 standby ACKs before confirming a config change to the operator. Operator changes cannot be silently lost on failover.

10. Hard SSL/cert gate: Controller refuses to boot if TLS cert/key files are missing or unreadable. Operator-friendly error message with file path.

11. Observable via logs: Election results, vote tallies, quorum counts, config tag comparisons, bootstrap operations, WAL commit quorum status, config push quorum status, and cert validation logged at INFO or WARN severity.

Approach

Replace LSF priority election with Raft-style quorum voting AND add majority-commit WAL replication for full CP durability. Config upgrades use the same majority-commit infrastructure for uniform CP semantics.

Eight focused changes:

Change A — Move listener, iomgr, and vtaskmgr before election

Extract bind()+listen(), iomgr_init(), and vtaskmgr_init() from cluster_init_phase2() so the full connection framework is running before election. vtaskmgr_init() uses the new --vcpus N CLI parameter (Change D item 1) instead of cluster_config->vcpus, removing the last config dependency on this startup block.

New startup order in myapp.c:

validate_peer_count()          ← Change D: odd-N check before any init
validate_certs()               ← Change D: cert pre-flight
cluster_init_phase1()          ← ha_init() reads ha-term.bin: restores term + voted_for
                               ← conf_read_tag() reads config_version_tag from disk (0 if absent)
                               ← Node now has durable identity before network is up
create_ssl_context()
iomgr_init() + iomgr_set_ssl_ctx()
cluster_open_listener(port)    ← bind + listen; register fd with iomgr
vtaskmgr_init(vcpus)
iomgr_start()
vtaskmgr_start()
atomic_store(ha.role, CTRL_INIT)    ← node now answers OP_requestVote + OP_probeConfigTag
bootstrap_config_if_needed()  ← Change C: if config_tag==0, probe peers; defers fetch until primary known
ha_wait_for_heartbeat_or_timeout()  ← Raft: sem_timedwait(election_timeout + rand_jitter)
                               ← heartbeat received → CTRL_STANDBY, skip election
                               ← timeout fires   → CTRL_ELECTING → am_i_primary_quorum()
[primary or standby path]
load_primary_config() / ha_connect_to_primary()
cluster_init_phase2_rest()    ← kvmgr, xstring, jobmgr, delaymgr, apiserver, etc.
cluster_run()

With iomgr and vtaskmgr running before election, incoming OP_requestVote messages are handled by the normal dispatch switch — no special pre-framework path needed. Outgoing vote requests use the existing connect_ssl_raw() + xce_encode()/xce_decode() pattern (same as config fetch), keeping per-peer threads lightweight.

Full 4-state controller FSM (ha_role_t):

The controller lifecycle is a single FSM stored in c->ha.role (atomic):

CTRL_INIT ──── timeout + jitter ──→ CTRL_ELECTING ──── quorum won ──→ CTRL_PRIMARY
          └─── heartbeat from        │                                     │
               existing primary ─→  └─── quorum lost ─→ CTRL_STANDBY ────┘
                                                              │
                                          EPOLLRDHUP / heartbeat timeout
                                                              │
                                                         CTRL_ELECTING

Dispatch gate (top of vtaskmgr switch, before any case):

switch (atomic_load(&c->ha.role)) {
case CTRL_INIT:
case CTRL_ELECTING:
    // Only vote + config-probe pass; all others get NOT_READY and retry (Change G)
    if (msg->op != OP_requestVote && msg->op != OP_probeConfigTag) {
        xce_send_error(task, XCE_ERR_NOT_READY, "controller not ready");
        return;
    }
    break;
case CTRL_STANDBY:
    // Primary-only ops rejected with XCE_ERR_STANDBY + redirect_host/port (Change G)
    break;
case CTRL_PRIMARY:
    break;  // full dispatch
}

State transition triggers:

• CTRL_INIT → CTRL_ELECTING: ha_wait_for_heartbeat_or_timeout() times out — no heartbeat received within election_timeout + rand_jitter. Different jitter per node (xorshift64 from jitter_seed) means nodes fire at staggered times, so the first candidate typically wins unopposed. This is Raft's primary split-vote prevention.
• CTRL_INIT → CTRL_STANDBY: heartbeat or OP_ctrlHello received from an existing primary during the wait window — node skips election entirely.
• CTRL_ELECTING → CTRL_PRIMARY: am_i_primary_quorum() returns 1.
• CTRL_ELECTING → CTRL_STANDBY: am_i_primary_quorum() returns 0.
• CTRL_STANDBY → CTRL_ELECTING (fast): EPOLLRDHUP on primary_ctrl_fd.
• CTRL_STANDBY → CTRL_ELECTING (fallback): heartbeat timeout vtask fires — primary is alive on TCP but not sending heartbeats. Vtask is reset on each heartbeat received; fires after election_timeout + rand_jitter of silence.

ha_wait_for_heartbeat_or_timeout() in myapp.c:

struct timespec dl = now_plus_ms(c->ha.config.election_timeout_ms +
                                  ha_rand_jitter(&c->ha));
int hb = (sem_timedwait(&c->ha.heartbeat_seen, &dl) == 0);
if (hb) {
    atomic_store(&c->ha.role, CTRL_STANDBY);
} else {
    atomic_store(&c->ha.role, CTRL_ELECTING);
    rc = am_i_primary_quorum(c);
}

ha.heartbeat_seen is posted by the heartbeat dispatch handler when any heartbeat or OP_ctrlHello is received while ha.role == CTRL_INIT.

Change B — Raft-style quorum voting in am_i_primary()

Replace the current timeout-based probe with a full request-vote protocol:

New wire message: OP_requestVote (Cap'n Proto)

struct RequestVoteRequest {
  term @0 :UInt64;              // candidate's term
  candidate_id @1 :Text;        // candidate's name (from --peers)
  config_tag @2 :UInt64;        // candidate's config SHA-256 tag (0 if none)
  last_log_lsn @3 :UInt64;      // candidate's WAL applied_lsn
  // NOTE: priority removed — Raft config_tag+LSN freshness fully determines winner;
  // jitter breaks split votes. Priority plays no role in quorum election.
}

struct RequestVoteReply {
  term @0 :UInt64;              // peer's current term
  vote_granted @1 :Bool;        // true = ACK
  peer_config_tag @2 :UInt64;   // peer's config tag (for config comparison)
  peer_last_log_lsn @3 :UInt64; // peer's WAL LSN (for log comparison)
}

Election uses iomgr + vtaskmgr (FSM model):

Since iomgr and vtaskmgr are running before election (Change A), vote requests are sent via iomgr's async connect path and replies arrive through the normal vtaskmgr dispatch switch. No election-specific threads are created.

New ha_vote_entry_t struct (ha.h) — per-peer election slice; only meaningful as ctx->votes[i] inside ha_elect_ctx_t:

typedef struct {
    xce_candidate_t *peer;
    uint64_t my_term;       // term sent in RequestVote
    uint64_t my_tag;        // config_tag sent in RequestVote
    uint64_t my_lsn;        // WAL LSN sent in RequestVote
    int      vote_granted;  // 1 = peer granted vote
    int      replied;       // 1 = peer responded (any reply, including denial)
    uint64_t peer_tag;      // peer's config_version_tag from reply
    uint64_t peer_lsn;      // peer's applied_lsn from reply
    // NOTE: no pthread_t — FSM uses iomgr async sends + vtaskmgr dispatch;
    //       ha_elect_ctx_t.done semaphore blocks am_i_primary_quorum()
} ha_vote_entry_t;

New ha_elect_ctx_t struct (ha.h) — holds all election state; single allocation via C99 flexible array member (one calloc, one free):

typedef struct {
    int              n_peers;
    int              votes_granted;  // includes self-vote (starts at 1)
    int              n_replied;      // peers that responded (any reply)
    uint64_t         max_resp_tag;   // freshest config_tag seen across all replies
    uint64_t         max_resp_lsn;   // freshest lsn seen across all replies
    int              result;         // set before sem_post: 1=primary, 0=standby, <0=error
    sem_t            done;           // startup thread blocks here; dispatch posts when done
    ha_vote_entry_t  votes[];        // C99 flexible array — n_peers-1 entries
} ha_elect_ctx_t;
// Allocate: calloc(1, sizeof(*ctx) + (n_peers-1)*sizeof(ctx->votes[0]))
// Free:     free(ctx)  — single allocation, no separate votes free needed

am_i_primary_quorum() (ha.c) — initiates election, blocks until done:

int am_i_primary_quorum(cluster_t *c) {
    int n_peers = c->pcfg.n_candidates;
    if (n_peers == 1) return 1;

    uint64_t my_term = atomic_fetch_add(&c->ha.term, 1) + 1;
    ha_persist_term(c, my_term, c->myconf->name);

    ha_elect_ctx_t *ctx = calloc(1, sizeof(*ctx) + (n_peers-1)*sizeof(ctx->votes[0]));
    ctx->n_peers       = n_peers;
    ctx->votes_granted = 1;   // self-vote
    ctx->max_resp_tag  = c->myconf->config_version_tag;
    ctx->max_resp_lsn  = atomic_load(&c->persist.applied_lsn);
    sem_init(&ctx->done, 0, 0);
    atomic_store(&c->ha.elect_ctx, ctx);  // dispatch reads this

    // Send RequestVote to all peers via iomgr (async non-blocking connects)
    for (int i = 0, j = 0; i < n_peers; i++) {
        if (strcmp(c->pcfg.candidates[i].name, c->myconf->name) == 0) continue;
        ctx->votes[j].peer   = &c->pcfg.candidates[i];
        ctx->votes[j].my_term = my_term;
        ctx->votes[j].my_tag  = c->myconf->config_version_tag;
        ctx->votes[j].my_lsn  = atomic_load(&c->persist.applied_lsn);
        iomgr_async_vote_request(c, &ctx->votes[j]);  // non-blocking
        j++;
    }

    // Arm election timeout vtask — fires ha_elect_timeout() if peers don't reply
    vtaskmgr_schedule(c, c->ha.config.election_timeout_ms, ha_elect_timeout, ctx);

    sem_wait(&ctx->done);  // startup blocks here; dispatch or timeout signals it

    int result = ctx->result;
    ha_elect_ctx_free(ctx);
    atomic_store(&c->ha.elect_ctx, NULL);
    return result;
}

handle_request_vote_reply() — called from vtaskmgr dispatch switch on OP_requestVoteReply; accumulates votes and signals completion:

void handle_request_vote_reply(cluster_t *c, RequestVoteReply *reply, int peer_idx) {
    ha_elect_ctx_t *ctx = atomic_load(&c->ha.elect_ctx);
    if (!ctx) return;  // no election in progress

    ha_vote_entry_t *v = &ctx->votes[peer_idx];
    v->replied = 1;
    v->peer_tag = reply->peer_config_tag;
    v->peer_lsn = reply->peer_last_log_lsn;
    if (reply->vote_granted) { v->vote_granted = 1; ctx->votes_granted++; }
    if (v->peer_tag > ctx->max_resp_tag ||
        (v->peer_tag == ctx->max_resp_tag && v->peer_lsn > ctx->max_resp_lsn)) {
        ctx->max_resp_tag = v->peer_tag; ctx->max_resp_lsn = v->peer_lsn;
    }
    ctx->n_replied++;

    int quorum = (ctx->n_peers + 1) / 2;
    // Signal when quorum achieved or all peers replied (or timed out)
    if (ctx->votes_granted >= quorum || ctx->n_replied == ctx->n_peers - 1)
        ha_elect_evaluate_and_signal(c, ctx);
}

ha_elect_timeout() vtask — fires when election_timeout_ms expires: counts unreplied peers as non-respondents, calls ha_elect_evaluate_and_signal().

ha_elect_evaluate_and_signal() — applies gates, sets ctx->result, signals:

• Defense-in-depth assertion (config_tag=0 BUG check)
• Hard gate B (yield if responding peer has fresher data → result=0)
• Quorum met → result=1
• Quorum lost → schedule jitter vtask, retry once with incremented term; if second round also fails → result=0; finally sem_post(&ctx->done)

Vote handler on peers (new function in ha.c):

void handle_request_vote(cluster_t *c, RequestVoteRequest *req, RequestVoteReply *reply) {
    pthread_mutex_lock(&c->ha.lock);
    uint64_t my_term = atomic_load(&c->ha.term);

    if (req->term < my_term) {        // stale candidate — reject
        reply->term = my_term;
        reply->vote_granted = 0;
        goto done;
    }
    if (req->term > my_term) {        // update term from candidate
        atomic_store(&c->ha.term, req->term);
        my_term = req->term;
        // Clear voted_for on new term transition
        c->ha.voted_for_term = 0;
        c->ha.voted_for_name[0] = '\0';
    }

    // Catch-up gate (Clarification 6): behind nodes must not influence elections
    if (c->ha.in_catch_up) {
        reply->vote_granted = 0;
        goto fill_reply;  // still send peer_config_tag/lsn so candidate has full picture
    }

    uint64_t my_tag = c->myconf->config_version_tag;
    uint64_t my_lsn = atomic_load(&c->persist.applied_lsn);
    int log_ok = (req->config_tag > my_tag) ||
                 (req->config_tag == my_tag && req->last_log_lsn >= my_lsn);

    // One vote per term: grant only if not yet voted, or voted for same candidate
    int already_voted = (c->ha.voted_for_term == my_term &&
                         strcmp(c->ha.voted_for_name, req->candidate_id) != 0);
    reply->vote_granted = (log_ok && !already_voted) ? 1 : 0;

    if (reply->vote_granted) {
        c->ha.voted_for_term = my_term;
        strncpy(c->ha.voted_for_name, req->candidate_id, XCE_MAXNAME_LEN - 1);
        ha_persist_term(c, my_term, req->candidate_id);  // durable before reply
    }
done:
    reply->term             = my_term;
    reply->peer_config_tag  = my_tag;
    reply->peer_last_log_lsn = my_lsn;
    pthread_mutex_unlock(&c->ha.lock);
}

Election safety extensions:

1. Term + voted_for persistence (ha-term.bin in {workdir}/): New ha_persist_term(cluster_t *c, uint64_t term, const char *voted_for) writes a 4-field struct atomically (write + rename) before any vote is granted or term incremented. On startup, ha_init() reads ha-term.bin and restores ha.term, ha.voted_for_term, ha.voted_for_name. Without this, a restarting node resets to term=0 and can double-vote in a term it already participated in.

2. New fields in ha_state_t (ha.h):

   uint64_t voted_for_term;                    // term of last granted vote
   char     voted_for_name[XCE_MAXNAME_LEN];   // candidate name of last granted vote
   int      primary_ctrl_fd;                   // persistent control channel fd (Change F; -1 = none)
   bool     defer_config_fetch;                // Change C: set when tag=0 and peers have config;
                                               // cleared after post-election fetch completes
   

3. Retire ha_check_priority_stepdown() (ha.c:322–368): This function's guard if (my_term > 0 || sender_term > 0) return makes it permanently unreachable after the first quorum election (term is always ≥ 1 thereafter). Change B explicitly removes the call site from the heartbeat handler and deletes the function body. Term-based leadership is fully handled by the reply.term > my_term check in handle_request_vote() and the existing ha_do_demote() path.

Key properties:

• Most up-to-date wins: A candidate that won quorum still yields if any responding peer has higher config_tag or (same tag, higher LSN). Non-responding peers never block election — only reachable peers with fresher state can cause a yield.
• Config freshness hard gate: Yielding due to fresher peer returns 0 (standby, not fatal). Only config_tag == 0 with peers having config_tag > 0 is fatal (-ENODATA), because that means the node has never been configured at all.
• Split vote prevention (primary): Each node waits election_timeout + rand_jitter before candidacy (Change A ha_wait_for_heartbeat_or_timeout()). Different jitter values stagger candidacy attempts so the first node typically wins unopposed — the same xorshift64 jitter_seed used elsewhere. This is Raft's fundamental split-vote prevention, not just recovery.
• Split vote recovery (safety net): If a split vote still occurs inside ha_elect_evaluate_and_signal(), a second attempt is scheduled with rand_jitter delay and incremented term. If that also fails, node returns 0 (standby). This safety net handles the rare case where multiple nodes have identical jitter values.
• No priority tiebreak: Priority is not used in Raft quorum election. When config_tag and LSN are identical across all peers, the jitter-based pre-election timeout is the tiebreaker. The ha.my_priority field is preserved in ha_state_t for backward compatibility but plays no role in election outcome.

Change C — Pre-election config bootstrap (deferred fetch)

A node with config_tag=0 is a config-seeker, not a candidate. Before calling am_i_primary_quorum(), bootstrap_config_if_needed() determines whether config exists among peers, but defers the actual fetch until the primary is known — avoiding a speculative download from a peer that may not win the election.

Rationale: Fetching from the highest-tag reachable peer before election can waste I/O: a fresher peer may start late, win the election, and push its config anyway. Since the election already proves who has the freshest state, bootstrap should defer to that proof rather than guessing.

New function bootstrap_config_if_needed() in config_repl.c (called from myapp.c):

void bootstrap_config_if_needed(cluster_t *c) {
    if (c->myconf->config_version_tag != 0) return;  // already have config

    int retry_ms = 500;
    for (;;) {
        uint64_t best_tag = 0;
        bool any_reachable = false;

        for (int i = 0; i < c->pcfg.n_candidates; i++) {
            if (strcmp(c->pcfg.candidates[i].name, c->myconf->name) == 0) continue;
            uint64_t peer_tag = 0;
            int rc = xce_probe_config_tag(c->pcfg.candidates[i].host,
                                          c->pcfg.candidates[i].port, &peer_tag);
            if (rc != 0) continue;   // peer unreachable — skip
            any_reachable = true;
            if (peer_tag > best_tag) best_tag = peer_tag;
        }

        if (!any_reachable) {
            WARN("No config and no peer reachable — retrying in %dms (kill to abort)",
                 retry_ms);
            usleep((useconds_t)retry_ms * 1000);
            retry_ms = MIN(retry_ms * 2, 10000);
            continue;
        }

        if (best_tag == 0) {
            INFO("All peers also have no config — first-boot cluster, proceeding");
            return;   // all-zero cluster: first boot, proceed to election as-is
        }

        // Config exists among peers — defer fetch until primary is known.
        // The election will prove who has the freshest state.
        INFO("Config exists among peers (best tag=%016" PRIx64 ") — "
             "deferring fetch until primary is known", best_tag);
        c->ha.defer_config_fetch = true;
        return;
    }
}

The function never returns an error. If no peer is reachable, it retries with exponential backoff. If all peers are tag=0, it returns for first-boot election. If config exists among peers, it sets defer_config_fetch and returns — the node will fetch from the proven winner after the election resolves.

xce_probe_config_tag() is a lightweight new RPC (single round-trip) that returns the peer's current config_version_tag without transferring the full config payload. Reuses the existing SSL connection setup pattern (connect_ssl_raw()).

Cap'n Proto schema for OP_probeConfigTag:

struct ProbeConfigTagRequest {
  name @0 :Text;   // requester's node name (for logging on peer side)
}
struct ProbeConfigTagReply {
  config_tag @0 :UInt64;  // peer's current config_version_tag (0 if no config)
}

Answered by all nodes in any FSM state (including CTRL_INIT) — the dispatch gate explicitly allows OP_probeConfigTag through before the controller is ready for normal requests (Change A). Note: the actual config transfer (OP_replConfig) does NOT need to be role-agnostic — with deferred fetch, do_one_config_fetch() is called post-election when the serving peer is already in CTRL_STANDBY or CTRL_PRIMARY and full dispatch is active. Only the probe op needs the CTRL_INIT exemption.

Integrated into myapp.c (before election):

bootstrap_config_if_needed(&c);  // returns immediately: tag>0, first-boot, or defer

// The tag=0 node participates in the election as a voter but never wins
// against nodes with tag>0 (log_ok gate ensures this). It votes, discovers
// the winner via voted_for, and fetches from the proven primary.
rc = am_i_primary_quorum(&c);
if (rc < 0) return rc;

if (c.ha.defer_config_fetch && rc != 1) {
    // Fetch from the proven winner (known via voted_for or heartbeat source).
    // Guard rc != 1: a winning node has tag>0 so defer_config_fetch is never
    // set for it, but the explicit guard prevents double load_primary_config()
    // if logic shifts during implementation.
    do_one_config_fetch(c->ha.primary_host, c->ha.primary_port, 0);
    load_primary_config(&c);
    c->ha.defer_config_fetch = false;
}

if (rc == 1) {
    rc = load_primary_config(&c);  // load into memory (already on disk)
    if (rc != 0) return rc;
    atomic_store(&c->ha.role, CTRL_PRIMARY);
}

Change D — Startup pre-flight checks + ASAN leak fix

main.c, src/controller/cluster.c, lib/netio.c, ha.c

1. --vcpus N CLI parameter (main.c, src/controller/cluster.c): Add a --vcpus N flag parsed in main.c alongside the existing --peers and --name flags. Default: 8 when not specified. Values < 1 are rejected at startup with "ERROR: --vcpus must be ≥ 1" and exit -EINVAL. Remove the vcpus field from cluster_config_t (currently read from the config file). Pass the parsed value directly to vtaskmgr_init() — this removes vtaskmgr_init()'s dependency on cluster_config, enabling it to run before election (Change A).

   Removed from cluster config schema: the vcpus: key in the config file is no longer read. Existing configs that include it receive a logged deprecation warning ("WARN: 'vcpus' in config file is deprecated — use --vcpus N instead") but the controller does not fail.

   Rationale: CPU count is a per-host capacity decision, not a cluster-wide policy. Different controller nodes in the same cluster may have different hardware. Keeping vcpus in the shared config forces all nodes to use the same value and couples an infrastructure tunable to the cluster state config file.

2. Peer count validation in main.c (before cluster_init_phase1()): The --peers list must contain exactly 1 node (single-node) or an odd number ≥ 3 (HA mode). Even numbers cannot form a strict majority and are rejected at startup:

   int n = pcfg.n_candidates;
   if (n != 1 && n % 2 == 0) {
       fprintf(stderr,
           "ERROR: --peers must list an odd number of controllers (1, 3, 5, ...).\n"
           "       %d controllers cannot form a majority. Use 1 for single-node\n"
           "       or add/remove one controller to reach an odd count.\n", n);
       return -EINVAL;
   }
   

   This removes the need for the n_peers == 2 special case inside am_i_primary_quorum() — the quorum formula (N+1)/2 is correct for all validated (odd) values of N.

3. Cert pre-flight in main.c (before myapp_run()): explicit access() check on {workdir}/certs/xce.key and {workdir}/certs/xce.crt. On failure: print operator-friendly "ERROR: TLS cert not found: <path>" and exit -ENOENT. Controller never starts without valid certs — same hard-prerequisite model as config.

4. Fix ASAN leak in ha_fini() (ha.c:451): add SSL_CTX_free(ha->ssl_ctx) call. This closes the ~5320-byte direct+indirect leak shown in the pc13v1 ASAN output.

Change E — Majority-commit WAL replication

src/controller/wal.c (or persist.c), src/controller/ha.c, src/controller/cluster.c

Principle: Primary does not ACK a write to its caller until ≥(N+1)/2 standbys have confirmed they wrote the WAL entry to disk. This guarantees any promoted standby has all committed entries — zero data loss on failover.

New wire message: OP_walAck (Cap'n Proto):

struct WalAckRequest {
  term     @0 :UInt64;   // primary's current term (for stale-detection)
  lsn      @1 :UInt64;   // WAL entry LSN being ACKed
}
struct WalAckReply {
  term     @0 :UInt64;   // standby's term (primary steps down if reply.term > mine)
  lsn      @1 :UInt64;   // highest LSN standby has durably written
  success  @2 :Bool;
}

Primary-side commit protocol (persist.c / wal.c):

1. Write WAL entry to local disk (existing)
2. Fan-out OP_walAck to all connected standbys in parallel (async send)
3. Wait for ≥(N+1)/2 − 1 standby ACKs (own write counts as 1) within commit_timeout_ms (configurable, default 5000ms)
4. On quorum ACK: mark entry committed, return success to caller
5. On timeout without quorum: return -ETIMEDOUT to caller; entry is written locally but not committed — next primary will re-evaluate on WAL replay

Standby-side WAL write (existing replication path extended):

• On receiving replicated WAL entry: write to disk, fsync, send OP_walAck reply
• No change to standby application logic

Backpressure: Primary maintains a per-standby pending_ack_count. If a standby falls more than max_pending_acks (default 64) entries behind, it is removed from the active replication set and reconnects via existing catch-up path.

Single-node mode: quorum = 1, commit is immediate (own write = majority). No change.

Clarification 1 — Durability boundary (which entry types block on majority-commit):

Not all WAL entry types have equal durability requirements. The rule:

Entry type	Commit model	Rationale
WAL_JOB_SUBMIT	Majority-commit (blocking)	Job loss on failover is visible to user
WAL_JOB_START / WAL_JOB_FINISH	Majority-commit (blocking)	State transitions must survive failover
WAL_QUEUE_CHANGE	Majority-commit (blocking)	Queue config changes must be durable
WAL_HOST_STATE	Best-effort async	Reconstructed from worker reconnect on failover
WAL_CHECKPOINT_REF	Best-effort async	Checkpoint is self-contained; ref is advisory

The existing persist_wal_sync_append() path (used for WAL_JOB_SUBMIT today with ANY-standby wait) is extended to use majority quorum. The existing persist_wal_async() path remains fire-and-forget for best-effort types. No new caller API — entry type determines which path is taken inside persist.c.

Clarification 2 — committed_lsn data model:

Two new fields added to persist_t (persist.h):

_Atomic uint64_t committed_lsn;  // highest LSN majority-ACKed; safe to serve/trim
uint64_t         applied_lsn;    // existing: highest LSN written locally (may exceed committed)

committed_lsn is advanced by repl_wait_majority_ack() after quorum is reached. On promotion, the new primary replays WAL from its checkpoint up to committed_lsn only — entries above committed_lsn are discarded (never committed to majority, so no client received success for them). WAL trim guard updated: trim threshold = committed_lsn, not applied_lsn.

Catch-up standbys (in_catch_up == 1) are excluded from quorum counting — they are not yet in the active replication set. This means a cluster with 2-of-3 nodes catching up cannot make progress until at least one catch-up completes. This is the correct CP behavior: prefer unavailability over serving uncommitted state.

Clarification 3 — repl_broadcast() prerequisite:

repl_broadcast() (repl.c:183–196) is currently a stub. Majority-commit requires a real broadcast implementation. This is co-developed with Change E — they are the same work unit. Change E scope explicitly includes:

• Implement repl_broadcast() with per-standby TCP send via existing iomgr pipeline
• Implement repl_wait_majority_ack() replacing the existing repl_wait_ack() (any→majority)
• Add per-standby send error callback: failed send → decrement active standby count, trigger catch-up reconnect (existing path)

If repl_broadcast() stub lands separately before Change E, Change E still works — it replaces the stub. No ordering dependency beyond Task 11 (config fetch).

New function persist_truncate_wal_above(persist_t *p, uint64_t cutoff_lsn) (persist.c):

Removes all WAL entries with LSN > cutoff_lsn from local storage. Called in two contexts (Clarifications 4 and 5). Implementation: scan WAL segment files from newest to oldest; truncate the file containing cutoff_lsn at the byte offset immediately after that entry; delete any segment files that start above cutoff_lsn. Updates p->applied_lsn = cutoff_lsn atomically after truncation. Must be idempotent (calling twice with same cutoff is safe). New function — does not currently exist in persist.c.

Clarification 4 — committed_lsn must be piggybacked on heartbeats:

A promoted standby needs to know where to truncate its WAL. Without this it only has applied_lsn, which may include uncommitted orphan entries written by the old primary but never majority-ACKed.

• Extend CtrlHeartbeat Cap'n Proto struct with committed_lsn @N :UInt64
• Standbys store it in ha.last_known_committed_lsn (_Atomic uint64_t, init 0) on each heartbeat receipt
• On promotion, immediately after am_i_primary_quorum() returns 1:

  persist_truncate_wal_above(&c->persist, c->ha.last_known_committed_lsn);
  c->persist.committed_lsn = c->ha.last_known_committed_lsn;
  

• First-boot failover (no heartbeat ever received): last_known_committed_lsn = 0, so WAL replay starts from checkpoint only — correct and safe

Clarification 5 — Log divergence on standby re-join:

The existing catch-up path assumes the standby is simply behind the primary (a clean subset). After a primary crash, a standby may have orphaned WAL entries above the new primary's committed_lsn — entries the old primary replicated but never majority-ACKed. Replaying those entries would corrupt the standby's state.

Catch-up handshake extension (Change E scope):

1. Primary sends its committed_lsn in the catch-up initiation reply
2. Standby calls persist_truncate_wal_above(&c->persist, primary_committed_lsn) before requesting WAL replay — discards any orphaned entries above that boundary
3. Standby sets applied_lsn = primary_committed_lsn as the new replay base
4. Normal WAL catch-up (existing mechanism) proceeds from there

Clarification 6 — Catch-up standbys excluded from voting (not just quorum count):

Clarification 2 excludes in_catch_up==1 standbys from the majority quorum count. They must also be prevented from granting votes — a behind-node's stale LSN could tip an election to a less-fresh candidate.

In handle_request_vote(), before the log_ok freshness check:

if (c->ha.in_catch_up) {
    reply->vote_granted = 0;
    // still populate peer_config_tag/peer_last_log_lsn — candidate needs the full picture
    goto fill_reply;
}

Clarification 7 — applied_lsn in vote request must be post-fsync:

RequestVoteRequest.last_log_lsn is sourced from atomic_load(&c->persist.applied_lsn). This is correct only if applied_lsn is advanced after fsync() completes in persist.c. A node that claims an LSN it has not durably written could win an election and then fail to replay entries it advertised. Verify in persist.c before implementing Change B — if applied_lsn is advanced pre-fsync anywhere, fix that first.

Change F — Persistent standby↔primary control channel

Files: src/controller/ha.c, src/controller/myapp.c, src/controller/cluster.c, src/controller/repl.c

Background: Standbys currently detect primary failure through ha_check_heartbeat_nocluster() (ha.c:380–405), which polls a last_heartbeat timestamp with a 10s floor. Failover detection latency is up to 10 seconds. Additionally, each config fetch and WAL replication message opens a new TCP+SSL connection (connect_ssl_raw(), config_repl.c:685–775), adding per-message handshake latency for WAL streaming (Change E).

Changes:

1. Standby discovers and connects to primary — new ha_find_and_connect_to_primary(cluster_t *c) (called in myapp.c standby path after election). Opens a TCP+SSL connection to the primary peer and registers the fd with epoll EPOLLIN | EPOLLRDHUP. The connection is stored in ha_state_t as ha->primary_ctrl_fd. On connect, the standby sends an extended OP_ctrlHello (item 5) carrying identity and state. The primary registers the standby, initiates catch-up, and begins streaming any missing WAL before the standby enters the active replication set.

   Primary discovery algorithm (how the standby finds the winner's address):

   Two startup paths lead here with different amounts of prior knowledge:

   • Heartbeat path (CTRL_INIT → CTRL_STANDBY): ha_wait_for_heartbeat_or_timeout() received a heartbeat/OP_ctrlHello from an existing primary. The sender's host/port is in the packet. Store immediately as ha.primary_host/port. No further discovery needed — connect directly.

   • Election path (CTRL_ELECTING → CTRL_STANDBY): lost am_i_primary_quorum(). Discovery uses a three-tier strategy exploiting Raft's structural property that the majority voted for the winner:

   Tier 1 (fast path — O(1) for ≥(N+1)/2 nodes):
     If ha.voted_for_name is set:
       try ha_connect_to_primary(voted_for peer)
       on success: store ha.primary_host/port, done
       on XCE_ERR_NOT_READY: peer still electing — backoff, retry tier 1
   
   Tier 2 (redirect — O(1) once any majority member connected):
     Probe remaining peers (random order) with OP_ctrlHello
     A CTRL_STANDBY peer that already knows the primary responds with
       XCE_ERR_STANDBY + redirect_host/port (same field as Change G)
     Follow redirect immediately — connect to that address
   
   Tier 3 (blind scan fallback — for split-vote minorities):
     If redirect not received: try remaining peers sequentially
     A CTRL_PRIMARY peer accepts OP_ctrlHello normally → connected
     A CTRL_INIT/CTRL_ELECTING peer returns XCE_ERR_NOT_READY → skip+retry
     Repeat with 100ms → 5s exponential backoff until primary found
   

   Why this converges quickly: In a 3-node cluster, at least 2 nodes voted for the winner. Those 2 connect via Tier 1 immediately. The 3rd probes either of them (Tier 2), gets a redirect, and connects. Total: 2 direct + 1 redirect = all standbys connected with at most one extra probe per minority node. In a 5-node cluster, at least 3 connect via Tier 1; the remaining 2 get redirected.

   ha.primary_host/port update: set on successful connect; cleared to empty on EPOLLRDHUP (primary disconnect). During the election window it is empty; standbys receiving OP_ctrlHello while ha.primary_host is empty reply with XCE_ERR_STANDBY and no redirect (graceful — probing node falls through to Tier 3).

   No new message types: the redirect uses the existing XCE_ERR_STANDBY error reply fields already extended in Change G (redirect_host, redirect_port). The standby dispatch gate (already returning XCE_ERR_STANDBY for primary-only ops) populates these fields from ha.primary_host/port when they are known.

2. Standby join synchronization — full state sync on connect (cluster.c, ha.c, repl.c):

   A standby with stale WAL cannot safely promote — its in-memory state (jobs, queues, resources) would be out of date, and promoted writes would replay entries clients already saw as committed. Every standby must be fully current before it enters the quorum.

   Extended OP_ctrlHello Cap'n Proto schema:

   struct CtrlHelloRequest {
     name        @0 :Text;    // peer name (from --peers)
     key         @1 :Data;    // cluster auth key: SHA-256(cluster_name + shared_secret)
                              // shared_secret read from myconf->cluster_key (new config field)
                              // primary rejects connections where HMAC does not match its own
     applied_lsn @2 :UInt64; // standby's current applied_lsn (0 if fresh node)
     config_tag  @3 :UInt64; // standby's current config_version_tag (0 if none)
   }
   struct CtrlHelloReply {
     committed_lsn  @0 :UInt64; // primary's current committed_lsn
     needs_snapshot @1 :Bool;   // true if standby's WAL predates oldest primary segment
     config_tag     @2 :UInt64; // primary's current config_version_tag
   }
   

   New wire messages for sync:

   struct SnapshotTransfer {
     snapshot_lsn @0 :UInt64; // LSN of this snapshot (= checkpoint_lsn after fresh checkpoint)
     payload      @1 :Data;   // serialized snapshot blob (from persist_checkpoint())
   }
   struct WalCatchupEntry {
     lsn     @0 :UInt64;      // WAL entry LSN
     type    @1 :UInt8;       // WAL entry type (WAL_JOB_SUBMIT etc.)
     payload @2 :Data;        // WAL entry payload
   }
   

   OP_snapshotTransfer and OP_walCatchupEntry flow on the persistent control channel fd — no new connections.

   Primary-side handle_ctrlhello() sequence (new handler in cluster.c):

   1. Extract name, applied_lsn, config_tag from CtrlHelloRequest
   2. repl_add_standby(&c->repl, ep, host, port)
   3. needs_snap = repl_needs_snapshot(&c->repl, applied_lsn, oldest_wal_lsn())
   4. Reply: CtrlHelloReply{c->persist.committed_lsn, needs_snap, myconf->config_version_tag}
   5. repl_start_catchup(&c->repl, ep, applied_lsn)   // sets standby's in_catch_up=1
   6. If needs_snapshot:
        persist_checkpoint(&c->persist)               // flush in-memory state to disk
        send OP_snapshotTransfer(snapshot_lsn, blob)
        applied_lsn = snapshot_lsn                    // advance standby's effective base
   7. Streaming loop — runs until standby is fully current:
        target_lsn = atomic_load(&c->persist.committed_lsn)   // read current live value
        stream WAL entries [applied_lsn+1 → target_lsn] via OP_walCatchupEntry
        wait for OP_walAck with acked_lsn >= target_lsn
        if atomic_load(&c->persist.committed_lsn) > target_lsn:
            applied_lsn = target_lsn; continue loop  // new entries committed, catch up again
        // standby is now at current committed_lsn — exit loop
   8. repl_complete_catchup(&c->repl, ep)   // in_catch_up=0; standby enters live replication
   9. If standby config_tag != primary config_tag:
        push_config_to_standbys(c, blob, blen, tag)
   

   repl_broadcast() skips in_catch_up==1 standbys: while the streaming loop (step 7) is in progress, repl_broadcast() does not send to this standby. The loop itself handles delivery of all committed entries — including entries committed after the handshake — by re-reading committed_lsn on each iteration. The loop exits only when the standby's acked_lsn equals the current committed_lsn, at which point repl_complete_catchup() fires and live broadcast takes over. There is no window where entries can be missed.

   Standby-side sequence (after receiving CtrlHelloReply in ha.c / myapp.c):

   1. Store primary_committed_lsn = reply.committed_lsn
   2. persist_truncate_wal_above(&c->persist, primary_committed_lsn)  // Clarification 5
   3. ha.last_known_committed_lsn = primary_committed_lsn             // Clarification 4
   4. ha.in_catch_up = 1                                              // deny votes during sync
   5. If reply.needs_snapshot:
        wait for OP_snapshotTransfer
        write received blob as local snapshot file
        load_snapshot(&c->persist, path, snapshot_lsn)
        applied_lsn = snapshot.snapshot_lsn
   6. Receive OP_walCatchupEntry stream; call apply_wal_entry() for each
   7. Send OP_walAck after each entry (or configurable batch):
        OP_walAck{acked_lsn = last applied LSN}
   8. When primary sends no more OP_walCatchupEntry (catch-up loop ended on primary side):
        ha.in_catch_up = 0   // standby now in live replication; votes granted normally
   

   apply_wal_entry() is the same WAL replay function used during startup recovery — it applies a single WAL entry to the in-memory state (jobs, queues, resources). Must exist or be extracted from the startup replay path before this change lands.

   repl_needs_snapshot() already exists in repl.c — compares standby_applied_lsn against oldest_wal_lsn (the trim floor). If standby is behind the trim floor, persist_checkpoint() is called on the primary to produce a fresh snapshot, which is then streamed in full. This is the only path where persist_checkpoint() is called on-demand rather than on schedule.

   push_config_to_standbys() is declared in config_repl.h; config sync at join handles the case where a standby was offline during a config upgrade.

3. Heartbeat runs on control channel — primary sends periodic heartbeats on the controller↔controller channel (in addition to existing worker heartbeats). Standby updates last_heartbeat on receipt, but the epoll path is the fast lane.

4. Replace timestamp polling with dual-trigger failover detection — ha_check_heartbeat_nocluster() (ha.c:380–405) is retired. Two complementary paths:

   • Fast path (EPOLLRDHUP): TCP disconnect detected immediately; primary_ctrl_fd closed and nulled before am_i_primary_quorum() is called. Sub-second latency.
   • Fallback path (heartbeat timeout vtask): armed after each heartbeat is received (vtaskmgr_schedule(election_timeout + rand_jitter, ha_heartbeat_timeout_cb)); reset on every subsequent heartbeat. Fires when primary is silent but TCP is alive — covers network partitions that keep TCP open, primary overload, or missed heartbeat bursts. On fire: close+null primary_ctrl_fd, set CTRL_ELECTING, call am_i_primary_quorum().

5. WAL streaming over persistent channel (Change E complement) — repl_broadcast() (repl.c:183–196) sends WAL entries on the persistent control channels established by each standby rather than opening per-entry connections. No new connection setup per WAL write.

Role flip on promotion: When a standby wins election and becomes primary, it must close and discard its primary_ctrl_fd (the old connection to the now-dead primary) before calling am_i_primary_quorum() triggers the role transition. After promotion, the new primary accepts OP_ctrlHello connections from the remaining standbys — the role of primary_ctrl_fd inverts from "my connection out" to "their connections in", tracked in a per-standby fd table rather than a single field.

Reconnect behavior: On primary_ctrl_fd disconnect, standby closes and nulls primary_ctrl_fd first, then calls am_i_primary_quorum(). If it becomes primary, the old primary_ctrl_fd is discarded. If it remains standby, it reconnects with exponential backoff (100ms, 200ms, 400ms, max 5s) until the new primary is reachable.

Single-node mode: ha_connect_to_primary() is a no-op when n_peers == 1.

Key properties:

• Sub-second failover detection (EPOLLRDHUP vs 10s timeout)
• No per-entry TCP+SSL handshake overhead for WAL replication
• Single liveness signal: connection alive = peer alive
• Backward-compatible: standby reconnect on disconnect triggers election (same semantics as today, faster latency)

Change G — libxce client-side retry for transient controller errors

Files: src/lib/ (libxce — client request/connection layer)

Background: With CTRL_ELECTING gating requests during the election window (Change A), and standbys already returning XCE_ERR_STANDBY to misdirected requests, every daemon (worker, scheduler) needs to handle these transient errors gracefully. Without built-in retry in libxce, each daemon would need its own ad-hoc loop — fragile and inconsistent.

Changes:

1. Retry policy in libxce's client request path (the function wrapping xce_encode() + send + recv + xce_decode()):

   • On XCE_ERR_NOT_READY: back off and retry against same endpoint
   • On XCE_ERR_STANDBY: redirect — the standby includes the current primary's host:port in the error payload when known (ha.primary_host/port). libxce reconnects directly to the redirected endpoint instead of blind backoff. When primary is unknown (e.g. during startup), falls back to normal retry.
   • On TCP-level ECONNREFUSED or ETIMEDOUT: reconnect and retry (handles the window before cluster_open_listener() runs on a starting peer)
   • Non-retriable errors (XCE_ERR_AUTH, XCE_ERR_INVALID, XCE_ERR_NOENT, XCE_ERR_PERM, etc.): surface immediately to caller

   XCE_ERR_STANDBY wire extension: add optional redirect_host (Text) and redirect_port (UInt16) fields to the error reply Cap'n Proto struct. Standbys populate these when ha.primary_host is non-empty; clients that don't understand the new fields simply retry with backoff (backward-compatible).

2. Retry configuration (sensible defaults, overridable per client):

   typedef struct {
       int     max_retries;          // default 20
       int     initial_retry_ms;     // default 50
       int     max_retry_ms;         // default 5000
       double  backoff_multiplier;   // default 2.0 (exponential)
   } xce_retry_config_t;
   

   Defaults cover a full election round (up to 20 × 50ms–5000ms) without requiring callers to configure anything. Daemons that need tighter or looser bounds pass a custom xce_retry_config_t.

3. Transparent to callers: existing daemon code makes one xce_request() call; libxce handles retry internally. No changes needed in worker, scheduler, or apiserver code.

Key properties:

• Decouples daemon startup order — a worker that starts before the primary is ready simply waits for the retry window to expire rather than crashing or looping manually
• XCE_ERR_STANDBY retry enables transparent primary failover: after failover, the next request to the old primary (now standby) gets XCE_ERR_STANDBY, libxce retries against the same host/port and the new primary responds
• Backoff prevents thundering-herd when 50 workers restart simultaneously after a failover

Change H — Majority-commit config push

src/controller/config_repl.c, src/controller/repl.c

Principle: Config upgrades (OP_upgradeConfig) use the same majority-commit infrastructure as WAL replication (Change E). The primary does not ACK a config change to the operator until ≥(N+1)/2 standbys have confirmed durable write.

Why this matters: Without majority-commit, a primary could accept a config change, ACK to the operator, then crash. The new primary serves the old config, and the operator's change is silently lost. For strict CP, config writes must have the same durability guarantee as WAL writes — even though they are orders of magnitude rarer.

Changes:

1. Reuse Change E infrastructure — repl_broadcast() (sends to all connected standbys on persistent channels) and repl_wait_majority_ack() (blocks until ≥(N+1)/2 ACKs or commit_timeout_ms) already built for WAL. Both functions accept a message-type parameter so they can serve WAL (OP_walAck) and config (OP_upgradeConfigAck) ACKs without duplication. Zero new infrastructure.

2. push_config_to_standbys() extended (config_repl.c):

   int push_config_to_standbys(cluster_t *c, const char *blob, size_t len, uint64_t tag) {
       // 1. Write locally (existing atomic write+rename — unchanged)
       int rc = config_repl_write_local(c, blob, len, tag);
       if (rc != 0) return rc;
   
       // 2. Fan-out OP_upgradeConfig via repl_broadcast (same channel as WAL)
       rc = repl_broadcast(c, OP_upgradeConfig, blob, len);
       if (rc != 0) return rc;
   
       // 3. Wait for ≥(N+1)/2 standby ACKs (own write = 1 vote)
       rc = repl_wait_majority_ack(c, c->ha.config.commit_timeout_ms);
   
       // 4. On quorum: advance committed_config_tag and return success
       if (rc == 0) {
           atomic_store(&c->myconf->committed_config_tag, tag);
       }
       return rc;  // -ETIMEDOUT on failure — operator retries
   }
   

3. Standby-side config ACK — on receiving OP_upgradeConfig, standby:

   • Writes config blob to disk (existing atomic write path)
   • Loads config into memory
   • Sends OP_upgradeConfigAck with acked_tag = written_tag
   • ACK semantics identical to OP_walAck — durable write complete

4. OP_upgradeConfigAck Cap'n Proto:

   struct UpgradeConfigAck {
     term @0 :UInt64;      // standby's term (primary steps down if ack.term > mine)
     acked_tag @1 :UInt64; // config tag standby has durably written
     success @2 :Bool;
   }
   

5. committed_config_tag field added to cluster config state (analogous to committed_lsn for WAL). Advanced only on majority ACK. Stored alongside config_version_tag in durable state. On promotion, a new primary serves only configs up to committed_config_tag. Piggybacked on heartbeats (same as committed_lsn — Clarification 4): standbys store the latest value in ha.last_known_committed_config_tag so a promoted standby knows the safe boundary without an extra round-trip. First-boot promotion (no heartbeat received): value is 0, standby serves whatever config it loaded from disk — correct for first-boot where no upgrade has been committed.

Key properties:

• Same commit_timeout_ms as WAL (configurable, default 5000ms)
• Single-node mode: quorum=1, commit is immediate (own write = majority)
• Config upgrades are operator-initiated and rare — the blocking ACK wait adds negligible overhead to cluster operations
• Catch-up standbys (in_catch_up==1) are excluded from quorum count, consistent with WAL majority rules
• If commit times out, operator gets -ETIMEDOUT and can retry — libxce retry (Change G) handles this transparently for the API server

Scope

• In:
  • Change A: Extract cluster_open_listener(), iomgr_init(), and vtaskmgr_init() from phase2; call all three before election in myapp.c so the full connection framework is running during election (incoming OP_requestVote handled by normal dispatch switch); add CTRL_ELECTING role + pre-dispatch gate returning XCE_ERR_NOT_READY for non-vote requests during election window
  • Change B: Full Raft-style quorum voting in am_i_primary_quorum()
    • New OP_requestVote wire protocol (Cap'n Proto request + reply)
    • handle_request_vote() handler on all peers with voted_for enforcement
    • Config tag + LSN freshness comparison in vote grant and candidate yield gate
    • Hard gate A (fatal): refuse primary if config_tag=0 and any peer has tag>0
    • Hard gate B (yield): winner yields if any responding peer has fresher data
    • Split vote retry with jitter (one retry after timeout)
    • Term + voted_for persistence to ha-term.bin (survives restart)
    • New ha_state_t fields: voted_for_term, voted_for_name, primary_ctrl_fd
    • Retire ha_check_priority_stepdown() (permanently unreachable after first election)
  • Change C: bootstrap_config_if_needed() pre-election bootstrap — probes peers for config tags, defers fetch until primary is known (avoids speculative download from a peer that may not win the election); tag=0 node participates as voter only; new xce_probe_config_tag() lightweight RPC; role-agnostic handle_config_request() so peers can serve config to a bootstrapping node
  • Change D: --vcpus N CLI parameter + remove vcpus from cluster config (deprecation warning for existing configs); odd-N peer-count validator at startup (-EINVAL on even count); cert pre-flight check in main.c; SSL_CTX_free ASAN fix in ha.c
  • Change E: Majority-commit WAL replication — OP_walAck protocol, primary commit gating, standby ACK path, backpressure, commit_timeout_ms config
  • Change F: Persistent standby↔primary control channel — ha_connect_to_primary(), EPOLLRDHUP-based failover detection, heartbeat on control channel, WAL streaming over persistent connection
  • Change G: libxce client-side retry — exponential backoff on XCE_ERR_NOT_READY, XCE_ERR_STANDBY, ECONNREFUSED, ETIMEDOUT; transparent to daemon callers; configurable via xce_retry_config_t with safe defaults
  • Change H: Majority-commit config push — OP_upgradeConfig uses the same repl_broadcast() + repl_wait_majority_ack() infrastructure as WAL replication (Change E). Primary waits for ≥(N+1)/2 standby ACKs before confirming a config upgrade to the operator. Reuses commit_timeout_ms. Zero new infrastructure cost.
  • Unit and integration tests for all eight changes
• Out:
  • Raft snapshot protocol — existing dual-zarray COW is orthogonal and unchanged
  • Joint consensus for cluster membership changes — candidate list is static
  • do_one_config_fetch() stub wiring (Task 11) — prerequisite; must land first or co-developed
  • mTLS between controllers — trusted-LAN model with server-side TLS is sufficient

Boundary Candidates

1. Election protocol — full replacement of am_i_primary() with quorum voting
2. Config wire protocol — new OP_requestVote, OP_probeConfigTag, OP_upgradeConfigAck; existing OP_replConfig used for post-election deferred fetch
3. Primary startup path — bootstrap integration in myapp.c
4. Listener lifecycle — extracted from phase2 to phase1

Out of Boundary

• Standby fetch path (fetch_config_from_primary()) — untouched
• Config format, versioning, or storage layout — untouched
• Raft snapshot protocol — existing dual-zarray COW is orthogonal
• Joint consensus for cluster membership changes — candidate list is static
• mTLS — trusted-LAN model with server-side TLS is sufficient

Upstream / Downstream

• Upstream: Task 11 (do_one_config_fetch() stub) on persistent_ha branch — with deferred fetch (Change C), the probe-only path in bootstrap_config_if_needed() does not call do_one_config_fetch(). Task 11 is hit only at the post-election deferred fetch step in myapp.c. If Task 11 is not yet landed, the deferred fetch returns -ENOSYS and the bootstrapping standby fails to load config — a fatal but clearly logged error. Both can be co-developed; Task 11 must land before or with Change C for a functional cluster.
• Downstream: Phase 3 hardening (membership changes, leader transfer), ops tooling for election/commit observability, monitoring metrics for WAL replication lag.

Existing Spec Touchpoints

• Adjacent: none of the active specs (resval-id-decoupling, scheduler-controller-msg-gap, worker-join-refactor) touch config replication, HA election, or cluster init.

Constraints

• C (gnu17), no C++, no new external dependencies.
• ASAN build must pass (nix build .#xcecore_debug).
• --peers must list 1 or an odd number ≥ 3 controllers. Even counts are rejected at startup with -EINVAL. This simplifies quorum math: (N+1)/2 is exact for all valid N; no special-case branches needed inside am_i_primary_quorum().
• Static membership: the candidate list is fixed at startup from --peers and never changes while the cluster is running. No dynamic join/leave of controller peers. Worker and scheduler nodes may still join dynamically — only controller peer membership is static. This eliminates the need for joint consensus (Raft §6) and locks the quorum denominator for the lifetime of the cluster.
• Single-node mode unaffected: quorum=1 everywhere (election, WAL commit = own write).
• TDD required: failing test first, then implementation.
• All new C test cases go in src/controller/testsuite/ before implementation.
• make test must pass without a live server.
• commit_timeout_ms must be configurable (default 5000ms) to allow tuning per deployment latency.
• WAL write stall during standby catch-up (operational constraint): In a 3-node cluster, quorum for WAL commit is 2 (primary + 1 standby). Between primary winning election and the first standby completing repl_complete_catchup(), the active replication set has 0 standbys. Every persist_wal_sync_append() call blocks for commit_timeout_ms and returns -ETIMEDOUT. This is CP-correct behavior — the cluster is unavailable until quorum is formed. Operators should expect a stall of up to commit_timeout_ms per write during the initial catch-up window on first boot or after a restart. Jobs submitted during this window receive a retriable error; libxce retry (Change G) handles this transparently for workers and schedulers.

Deployment Reference

Controller node topology

Cluster size	Failure tolerance	WAL ACKs needed	Recommendation
1 node	0	self only	dev/test only
3 nodes	1 failure	1 standby ACK	minimum for production
5 nodes	2 failures	2 standby ACKs	recommended for critical HPC
7 nodes	3 failures	3 standby ACKs	rarely justified; adds latency

Job submission throughput (Raft overhead only)

Basis: LAN environment, 5 controller nodes, quorum = 3 (primary + 2 standby ACKs).

Critical path per WAL_JOB_SUBMIT:

client → primary:        RTT/2 + primary WAL fsync
primary → 2 standbys:   RTT/2 + standby WAL fsync  (parallel, overlaps with above)
2nd ACK → primary:      RTT/2
primary confirms:        RTT/2
─────────────────────────────────────────────────
Total:  1 LAN RTT + max(fsync_primary, fsync_standby)

Parameter	Optimistic	Conservative
LAN RTT (controller↔controller)	0.1 ms	0.5 ms
NVMe fsync latency	0.05 ms	0.2 ms
Effective commit latency	~0.2 ms	~0.7 ms

Throughput estimates:

Submit model	Commit latency	Jobs/sec
Serial single stream	0.3–1 ms	1,000–3,300
10 concurrent clients	0.5 ms	~20,000
100 concurrent clients	0.5 ms	~200,000 (non-Raft bottlenecks dominate before this)

Practical HPC ceiling: 5,000–20,000 jobs/sec before jobmgr queue locking and in-memory state mutations become the constraint.

5-node vs 3-node write latency: negligible. Majority-commit waits for the 2nd-fastest ACK out of 4 standbys (5-node) vs the 1st out of 2 (3-node). Under uniform LAN latency the difference is sub-millisecond. The extra two nodes cost nothing in write throughput but double the failure tolerance.

When Raft becomes a bottleneck: automated pipelines submitting >10,000 short jobs/sec. Mitigation: batch multiple job submits into a single WAL entry at the API layer before this design makes that necessary.