myxce/.kiro/specs/election-iomgr-refactor/design.md

Design Document: election-iomgr-refactor

Overview

The XCE controller was originally fully event-driven: cluster_init → cluster_start → cluster_run (shutdown-check loop). The HA additions broke this by inserting blocking calls before cluster_run: ha_wait_for_heartbeat_or_timeout and am_i_primary_quorum block the main thread, causing SSL context confusion, capnp UAF, and mixed-I/O races.

This design restores full event-driven operation with no blocking calls after phase1. myapp_run does setup then immediately enters cluster_run. All election I/O flows through iomgr callbacks. When the election reaches a terminal state, ha_elect_evaluate_and_signal calls cluster_init_phase2 inline from vtask context (phase2 is already thin — managers were moved to phase1 in a prior refactor), then atomically sets ha->role. The dispatch gate (ha_dispatch_gate) opens automatically: the next client request sees PRIMARY or STANDBY and is handled normally. No semaphores, no wbuf signals, no changes to cluster_run.

Goals

• Restore event-driven model: myapp_run → cluster_run with zero blocking between them
• All election I/O through one epoll event loop (iomgr)
• ha_elect_evaluate_and_signal calls thin cluster_init_phase2 from vtask context; sets role atomically
• cluster_run unchanged — still just the shutdown-check loop
• myreq_handler / ha_dispatch_gate is the sole FSM dispatcher — no new state machinery needed
• Role-based vote gate: PRIMARY and STANDBY-with-live-primary deny votes

Non-Goals

• Changes to cluster_run or wbuf signal values
• Incoming vote RPC handling (already correct in cluster.c)
• Heartbeat watchdog thread (existing, unchanged)
• Config replication, RDHUP paths (unchanged)
• Async DNS resolution (deferred)
• TLS certificate verification for vote connections

Boundary Commitments

This Spec Owns

• vote_request_thread + am_i_primary_quorum blocking loop → replaced by ha_start_election_async()
• on_connected callback added to netio.c
• client_ssl_ctx in ha_state_t + myapp_run
• vote_reply_msg_handler for iomgr-native reply handling (Note: this handler was later generalized into peer_ctrl_msg_handler, a single state-aware dispatcher that also routes repl acks and PreVote replies — superseded by ha-prevote, see that spec's design.md)
• Election timeout watchdog thread
• ha_elect_evaluate_and_signal extended to call cluster_init_phase2 then set role
• handle_request_vote extended with role-based deny gate (Note: handle_request_vote is now #ifdef XCE_TESTING-only; production vote dispatch is cluster.c's case OPCode_requestVote: calling raft_on_vote_request() — superseded by raft-adapter-conformance, see that spec's design.md)
• myapp_run restructured: phase1 + SSL + ha_start_election_async + cluster_run directly

Out of Boundary

• cluster_run internals (unchanged)
• case OPCode_requestVote: in cluster.c (already correct, no changes)
• ha_wait_for_heartbeat_or_timeout (existing; now called internally by ha_start_election_async)
• Config replication SSL

Allowed Dependencies

• netio.c / endpoint_t (add on_connected callback)
• ha_elect_ctx_t (existing; mutex guards concurrent access)
• iomgr_t / iomgr_monitor (existing)
• xce_request() / endpoint_enq() (existing)
• handle_request_vote_reply() (existing)
• cluster_init_phase2() (called from vtask context after election; must be vtask-safe)

Revalidation Triggers

• Changes to endpoint_t structure
• Changes to cluster_init_phase2 preconditions (anything it accesses that may not be ready)
• Changes to ha_dispatch_gate CTRL_ELECTING allowlist

Architecture

Before: Blocking startup

graph TD
    myapp_run --> cluster_init_phase1
    cluster_init_phase1 --> ha_wait_hb_or_timeout
    ha_wait_hb_or_timeout -->|blocks 2s| timeout
    timeout --> am_i_primary_quorum
    am_i_primary_quorum -->|pthreads block on sem| result
    result --> cluster_init_phase2
    cluster_init_phase2 --> cluster_start
    cluster_start --> cluster_run

After: Fully event-driven

graph TD
    myapp_run --> cluster_init_phase1
    cluster_init_phase1 --> ssl_setup
    ssl_setup --> cluster_register_listener
    cluster_register_listener --> iomgr_vtask_start
    iomgr_vtask_start --> ha_start_election_async
    ha_start_election_async -->|returns immediately| cluster_run

    iomgr_worker -->|EPOLLOUT handshake| on_vote_connected
    on_vote_connected --> xce_request_vote
    iomgr_worker -->|EPOLLIN reply| vtask_vote_reply_msg_handler
    vtask_vote_reply_msg_handler --> handle_request_vote_reply
    handle_request_vote_reply --> ha_elect_evaluate_and_signal
    ha_elect_evaluate_and_signal --> cluster_init_phase2_thin
    cluster_init_phase2_thin --> set_ha_role_atomic
    set_ha_role_atomic -->|gate opens| serve_traffic

    election_watchdog -->|timeout| ha_elect_evaluate_and_signal
    heartbeat_watchdog -->|hb timeout or received| ha_elect_evaluate_and_signal

Key Decisions

• ha_start_election_async() is fully non-blocking: Arms heartbeat watchdog + election timeout watchdog, connects outgoing vote endpoints, registers them with iomgr. Returns immediately. (Note: peer vote-endpoint connection and vote tallying are no longer driven directly by this function — they are now owned by the Raft library (src/lib/raft/raft.c): ha_start_election_async() calls raft_start_election(), which fires raft_cb_send_vote_request per peer and raft_cb_become_leader/raft_cb_become_follower on a terminal result. Superseded by raft-log-replication and raft-commit-safety, see those specs' design.md.)
• Two promotion paths in ha_elect_evaluate_and_signal:
  • Initial startup (ha->phase2_done == 0): calls cluster_init_phase2 (loads config data, inits structures), sets ha->phase2_done = 1. Phase2 runs exactly once per process lifetime — config upgrade always does a full restart.
  • RDHUP failover (ha->phase2_done == 1): state is current via continuous WAL replication. Skip phase2; enter COMMITTING directly. (Note: see CTRL_PRIMARY_COMMITTING note below — this now enters CTRL_PRIMARY directly, no COMMITTING state.)
• is_first_boot = ha->phase2_done == 0: Not !c->running (which is 0 until cluster_run sets it, making all elections look like first boot). phase2_done is a dedicated bool in ha_state_t. (Still accurate — verified current in ha.c.)
• CTRL_PRIMARY_COMMITTING: New FSM state between winning election and serving traffic. Allows repl acks; denies job dispatch. Also denies vote grants (challenger must not displace a committing primary). (Note: superseded by ha-single-leader-role, see that spec's design.md — CTRL_PRIMARY_COMMITTING was subsequently removed. ctrl_role_t now has only 4 values (CTRL_PRIMARY, CTRL_STANDBY, CTRL_INIT, CTRL_ELECTING); a winning candidate enters CTRL_PRIMARY directly via raft_cb_become_leader/ha_elect_evaluate_and_signal, with no transitional committing state and no committing-timeout watchdog. Job-traffic admission during startup is now gated by ha_has_write_quorum() inside the CTRL_PRIMARY case of ha_dispatch_gate, not by a separate FSM state. Every other reference to CTRL_PRIMARY_COMMITTING in this document — the sequence diagrams, the ha_repl_ack_received()/ha_dispatch_gate COMMITTING-case code samples, the Error Handling and Testing Strategy sections below — describes this now-removed mechanism.)
• cluster_run unchanged: Still just cbuf_get(&wbuf) for stop signal.
• delaymgr_start and persist_start move to myapp_run: Currently called inside cluster_start(). In the new design cluster_start is not called from myapp_run. These two calls move explicitly into myapp_run before ha_start_election_async.
• bootstrap_config_if_needed removed entirely: Replaced by the freshness gate in voting + primary replication push. No pre-election config probe needed.
• ha_send_ctrlhello_to_all_standbys: Iterates ha->standby_endpoints (populated when standbys connect and send OPCode_hello), sends CtrlHelloRequest to each. This triggers the existing catch-up replication in handle_ctrlhello. After catch-up, standby sends OPCode_replCatchUpDone → ha_repl_ack_received.
• committing_timeout_ms: Added to ha_config_t with default HA_DEFAULT_COMMITTING_TIMEOUT_MS = 10000 (10s). Covers the case where standbys are down at startup (single-node degraded mode).
• Timeout mechanism — delay_task_ms + cancelled flag: delaymgr is enhanced with one new API:
  • delay_task_ms(mgr, delay_ms, runcount, fn, arg) — millisecond-granularity one-shot task using clock_gettime(CLOCK_MONOTONIC). Precision bounded by mgr->precision_ms.
  • Cancellation via state flag: each context struct (election_timeout_ctx_t, committing_timeout_ctx_t) carries an _Atomic int cancelled field. Setting it to 1 makes the callback a no-op when it fires. This is O(1), lock-free, and race-free — even if the worker dequeued the task and is mid-execution, the flag check at callback entry handles it correctly. No ordq scan or ID tracking needed.

  typedef struct {
      _Atomic int    cancelled;
      ha_state_t    *ha;
      cluster_t     *c;
      ha_elect_ctx_t *ctx;
  } election_timeout_ctx_t;
  
  static int election_timeout_fn(void *arg) {
      election_timeout_ctx_t *t = arg;
      if (atomic_load(&t->cancelled)) return 0;
      ha_elect_evaluate_and_signal(t->ha, t->ctx);
      return 0;
  }
  
  /* Schedule: */
  delay_task_ms(&c->delaymgr, ha->config.election_timeout_ms, 1, election_timeout_fn, ctx);
  
  /* "Cancel" when quorum already reached: */
  atomic_store(&ctx->cancelled, 1);
  

  The existing heartbeat watchdog keeps its dedicated sem_timedwait pthread (precise rearm on every heartbeat — not suitable for delaymgr's polling model).

File Structure Plan

src/lib/
├── netio.c              # Add on_connected callback to endpoint_t; invoke in io_handler
├── delay.c              # Add delay_task_ms() — ms-granularity using clock_gettime(CLOCK_MONOTONIC)
│                        # Cancellation via caller-owned _Atomic int cancelled flag (no API needed)
└── delay.h              # Expose delay_task_ms()

src/controller/
├── ha.h                 # Add to ha_state_t: client_ssl_ctx, phase2_done, repl_ack_count, n_peers
│                        # Add to ha_config_t: committing_timeout_ms (default 10000ms)
│                        # Add CTRL_PRIMARY_COMMITTING to ctrl_role_t enum
│                        # (Note: repl_ack_count, committing_timeout_ms, and CTRL_PRIMARY_COMMITTING
│                        #  do not exist in current ha.h — superseded by ha-single-leader-role,
│                        #  see that spec's design.md. client_ssl_ctx, phase2_done, n_peers verified current.)
│                        # Declare: ha_start_election_async(), ha_election_cleanup(),
│                        #          ha_repl_ack_received(), ha_arm/disarm_committing_timeout(),
│                        #          ha_send_ctrlhello_to_all_standbys()
│                        # (Note: ha_repl_ack_received/ha_arm/disarm_committing_timeout do not
│                        #  exist — superseded by ha-single-leader-role. ha_send_ctrlhello_to_all_standbys
│                        #  verified current, called from raft_cb_become_leader.)
├── ha.c                 # Remove: vote_request_thread, vote_thread_ctx_t
│                        # Add: vote_ep_ctx_t, on_vote_connected(), vote_reply_msg_handler()
│                        # (Note: vote_reply_msg_handler was generalized into peer_ctrl_msg_handler —
│                        #  superseded by ha-prevote, see that spec's design.md)
│                        # Add: ha_start_election_async(), election_timeout_watchdog
│                        # Extend ha_elect_evaluate_and_signal: COMMITTING path + replication push
│                        # Extend handle_request_vote: role-based deny gate
│                        # (Note: handle_request_vote is now #ifdef XCE_TESTING-only — superseded
│                        #  by raft-adapter-conformance, see that spec's design.md)
│                        # Add: ha_repl_ack_received(), ha_arm/disarm_committing_timeout()
│                        # Extend ha_dispatch_gate: CTRL_PRIMARY_COMMITTING case
│                        # (Note: none of these three exist — superseded by ha-single-leader-role;
│                        #  see the CTRL_PRIMARY_COMMITTING note in Key Decisions above)
├── cluster.c            # cluster_init_phase2: vtask-safe (managers already in phase1)
│                        # ha_send_ctrlhello_to_all_standbys() or reuse handle_ctrlhello path
│                        # cluster_run: UNCHANGED
└── myapp.c              # Restructured: phase1 + SSL + delaymgr/persist_start + ha_start_election_async + cluster_run
                         # Remove: ha_wait_for_heartbeat_or_timeout, am_i_primary_quorum,
                         #         for(;;) retry loop, cluster_init_phase2, cluster_start calls,
                         #         bootstrap_config_if_needed (replaced by freshness gate + primary push)

Modified Files

• src/lib/netio.c — Add on_connected to endpoint_t; fire in io_handler after ssl_handshake_done=1
• src/lib/delay.c — Add delay_task_ms() using clock_gettime(CLOCK_MONOTONIC) for ms precision; precision still bounded by mgr->precision_ms
• src/lib/delay.h — Expose delay_task_ms()
• src/controller/ha.h — Add client_ssl_ctx, phase2_done, repl_ack_count, CTRL_PRIMARY_COMMITTING; declare new functions
• src/controller/ha.c — Main changes; ha_dispatch_gate extended for COMMITTING; uses delay_task_ms/delay_task_cancel for watchdogs
• src/controller/cluster.c — ha_send_ctrlhello_to_all_standbys helper; OPCode_replCatchUpDone calls ha_repl_ack_received; cluster_run unchanged
• src/controller/myapp.c — Restructured: non-blocking after phase1; delaymgr_start, persist_start moved here; bootstrap_config_if_needed removed

System Flows

3-node startup — ctrl-a wins (with COMMITTING phase)

sequenceDiagram
    participant Main as myapp_run
    participant Async as ha_start_election_async
    participant iomgr as iomgr Worker
    participant vtask as vtaskmgr Worker
    participant PeerB as ctrl-b endpoint
    participant PeerC as ctrl-c endpoint

    Main->>Main: cluster_init_phase1 + SSL + register_listener + delaymgr_start + persist_start
    Main->>Async: ha_start_election_async (non-blocking)
    Async->>PeerB: endpoint_connect(client_ssl_ctx)
    Async->>PeerC: endpoint_connect(client_ssl_ctx)
    Async->>iomgr: iomgr_monitor(peer_eps)
    Async-->>Main: returns immediately

    Main->>Main: cluster_run() — wbuf shutdown loop

    iomgr->>PeerB: handshake → on_vote_connected → OP_requestVote
    iomgr->>PeerC: handshake → on_vote_connected → OP_requestVote
    iomgr->>vtask: vote replies → quorum reached
    vtask->>vtask: ha_elect_evaluate_and_signal → cluster_init_phase2 (first boot)
    vtask->>vtask: ha->role = CTRL_PRIMARY_COMMITTING
    vtask->>PeerB: push_config_to_standbys + ha_send_ctrlhello (triggers WAL catch-up)
    vtask->>PeerC: push_config_to_standbys + ha_send_ctrlhello

    iomgr->>vtask: OPCode_replCatchUpDone from ctrl-b → ha_repl_ack_received (ack=1)
    iomgr->>vtask: OPCode_replCatchUpDone from ctrl-c → ha_repl_ack_received (ack=2 ≥ quorum-1)
    vtask->>vtask: ha_promote_to_primary → ha->role = CTRL_PRIMARY
    Note over vtask: dispatch gate fully open — serves job traffic

Primary crash — heartbeat timeout triggers promotion (RDHUP failover)

sequenceDiagram
    participant HBWatch as Heartbeat Watchdog on ctrl-b
    participant vtask as vtaskmgr Worker
    participant iomgr as iomgr on ctrl-b
    participant PeerC as ctrl-c endpoint

    Note over HBWatch: heartbeat_timeout_ms — primary silent
    HBWatch->>vtask: ha_on_primary_ctrl_fd_rdhup → ha->role = CTRL_ELECTING
    HBWatch->>iomgr: ha_start_election_async (retry — is_first_boot=false)
    iomgr->>PeerC: vote exchange → ctrl-b wins
    vtask->>vtask: ha_elect_evaluate_and_signal (is_first_boot=false → skip phase2)
    vtask->>vtask: ha->role = CTRL_PRIMARY_COMMITTING
    vtask->>PeerC: ha_send_ctrlhello (WAL already current — catch-up is fast)
    iomgr->>vtask: OPCode_replCatchUpDone from ctrl-c → ha_repl_ack_received (ack=1 ≥ quorum-1)
    vtask->>vtask: ha_promote_to_primary → ha->role = CTRL_PRIMARY
    Note over vtask: ctrl-b is new primary — state already synchronized via prior replication

Incoming request during election (NOT_READY)

sequenceDiagram
    participant Client
    participant iomgr as iomgr Worker
    participant vtask as vtaskmgr Worker

    Client->>iomgr: TCP connect + OP_jobsubmit
    iomgr->>vtask: myreq_handler
    vtask->>vtask: ha_dispatch_gate(CTRL_ELECTING, OPCode_jobsubmit) → XCE_ERR_NOT_READY
    vtask->>Client: reply status = XCE_ERR_NOT_READY (client retries)

Requirements Traceability

Requirement	Summary	Components	Flows
1.1	No pthreads for vote I/O	ha_start_election_async	3-node startup
1.2	Vote RPC via iomgr path	on_vote_connected, xce_request	3-node startup
1.3	Non-responding peer → deny	election watchdog → evaluate	—
2.1	Async TCP+TLS via iomgr	endpoint_connect, iomgr_monitor	3-node startup
2.2	TLS client context	client_ssl_ctx in ha_state_t	3-node startup
2.3	Send vote after handshake	on_vote_connected callback	3-node startup
2.4	Connect/TLS failure → deny	vote_ep_err_handler	—
3.1	Reply via iomgr packet	vote_reply_msg_handler§	3-node startup
3.2	No capnp UAF	vtask handler reads before free	3-node startup
3.3	Extract reply fields	vote_reply_msg_handler§ decodes	3-node startup
3.4	Term fencing on reply	handle_request_vote_reply (existing)	—
4.1	Quorum → CTRL_PRIMARY_COMMITTING†	ha_elect_evaluate_and_signal → committing state†	3-node startup
4.2	Freshness gate guarantees ground-truth	handle_request_vote config_tag check‡	—
4.3	Cold start: only highest config_tag wins	freshness gate + role gate combined	—
4.4–4.7	Retry, timeout, heartbeat abort	watchdog, re-arm, heartbeat watchdog	Primary crash
5.1	ELECTING gate: NOT_READY for non-election	ha_dispatch_gate (existing + COMMITTING)†	NOT_READY flow
5.2	COMMITTING: only repl acks allowed	ha_dispatch_gate new COMMITTING case†	3-node startup
5.3	PRIMARY: full traffic	ha_dispatch_gate (existing)	—
5.4–5.5	STANDBY gate, term persistence	ha_dispatch_gate, ha_persist_term (existing)	—
6.1	PRIMARY denies all votes	handle_request_vote role gate‡	—
6.2	STANDBY+fresh hb denies votes	handle_request_vote heartbeat check‡	—
6.3	HB timeout → ELECTING	heartbeat watchdog fires RDHUP path	Primary crash
6.4	INIT/ELECTING evaluates Raft gates	handle_request_vote existing gates‡	—
6.5	Primary crash → new primary	HB timeout → re-election → COMMITTING → PRIMARY†	Primary crash
7.1	Winner pushes config to standbys	push_config_to_standbys() in committing path	3-node startup
7.2	Winner replicates WAL to standbys	handle_ctrlhello() catch-up (existing)	3-node startup
7.3	config_tag=0 standby gets config via push	no independent fetch; primary pushes	3-node startup
7.4	Ack tracked per standby	ha_state_t.repl_ack_count†	3-node startup
8.1	COMMITTING blocks job traffic	ha_dispatch_gate COMMITTING case†	3-node startup
8.2	Majority ack → CTRL_PRIMARY	ha_repl_ack_received() threshold check†	3-node startup
8.3	CTRL_PRIMARY opens full traffic	ha_dispatch_gate PRIMARY case (existing)	3-node startup
8.4	Ack timeout → degraded open	timeout watchdog in committing state†	—
9.1–9.4	Regressions	Single-node, ground-truth invariant, RDHUP hooks-only, priority stagger	—

† Rows marked † describe the CTRL_PRIMARY_COMMITTING mechanism, removed by ha-single-leader-role — see the Key Decisions note above (a winner now enters CTRL_PRIMARY directly; job-traffic admission during startup is gated by ha_has_write_quorum() inside ha_dispatch_gate's CTRL_PRIMARY case). ‡ Rows marked ‡ cite handle_request_vote, now #ifdef XCE_TESTING-only; production vote-request handling is cluster.c's case OPCode_requestVote: → raft_on_vote_request() (superseded by raft-adapter-conformance, see that spec's design.md). § Rows marked § cite vote_reply_msg_handler, generalized into peer_ctrl_msg_handler (superseded by ha-prevote, see that spec's design.md).

Components and Interfaces

Summary

Component	File	Intent	Req Coverage
endpoint_t.on_connected	netio.c	Async handshake notification	2.1, 2.3
ha_state_t.client_ssl_ctx	ha.h/myapp.c	TLS client context	2.2
vote_ep_ctx_t	ha.c	Per-peer context for callbacks	1.1, 2.1–2.4
on_vote_connected()	ha.c	Send vote RPC after handshake	2.3, 3.1
vote_reply_msg_handler()§	ha.c	Decode reply; update tally	3.1–3.4
ha_start_election_async()¶	ha.c	Non-blocking: connect peers + arm watchdog	1.1–1.3, 4.1–4.3
Election timeout watchdog	ha.c	Fire evaluate if deadline expires	1.3, 4.2, 4.3
handle_request_vote role gate‡	ha.c	Deny votes from PRIMARY/healthy-STANDBY	6.1–6.4
ha_elect_evaluate_and_signal extended	ha.c	Enter COMMITTING; push config+WAL; set role†	4.1, 7.1–7.4
CTRL_PRIMARY_COMMITTING state†	ha.h + ha_dispatch_gate	Block job traffic; allow repl acks	5.2, 8.1
ha_repl_ack_received()†	ha.c	Count standby acks; transition to PRIMARY on quorum	8.2–8.3
ha_state_t.repl_ack_count†	ha.h	Atomic ack counter per election round	7.4, 8.2

† = superseded by ha-single-leader-role (CTRL_PRIMARY_COMMITTING removed; see Key Decisions note above). ‡ = handle_request_vote is now test-only; production path is raft_on_vote_request() (raft-adapter-conformance). § = renamed/generalized to peer_ctrl_msg_handler (ha-prevote). ¶ = peer connect + vote tally now delegated to raft_start_election()/raft_on_vote_reply() (raft-log-replication, raft-commit-safety).

Key Interfaces

ha_start_election_async()

(Note: superseded by raft-log-replication and raft-commit-safety, see those specs' design.md. This function no longer connects vote endpoints or tallies votes directly — it delegates to raft_start_election(), which fires raft_cb_send_vote_request per peer and, on a terminal result, raft_cb_become_leader/raft_cb_become_follower. The signature and non-blocking contract below remain accurate.)

/* Non-blocking. Arms heartbeat watchdog + election watchdog.
 * Connects outgoing vote endpoints. On terminal state:
 * ha_elect_evaluate_and_signal() calls cluster_init_phase2() then sets ha->role. */
void ha_start_election_async(ha_state_t      *ha,
                             cluster_t        *c,
                             int               n_peers,
                             xce_candidate_t **peers,
                             uint64_t          my_config_tag,
                             uint64_t          my_lsn,
                             const char       *my_name);

ha_elect_evaluate_and_signal (extended)

(Note: superseded by ha-single-leader-role, see that spec's design.md. The COMMITTING transition below — CTRL_PRIMARY_COMMITTING, repl_ack_count, ha_arm_committing_timeout — no longer exists; a winning candidate is set to CTRL_PRIMARY directly. The is_first_boot/phase2_done gating and the STANDBY branch below remain accurate and current.)

/* After election determines role: */
int is_first_boot = !ha->phase2_done;  /* phase2_done set after first cluster_init_phase2 call */

if (result == CTRL_PRIMARY) {
    if (is_first_boot) {
        cluster_init_phase2(c);    /* load config+data; runs once per process lifetime */
        ha->phase2_done = 1;
    }
    /* Enter COMMITTING: enable hooks but block job traffic until majority ack (Req 8.1) */
    atomic_store(&ha->role, CTRL_PRIMARY_COMMITTING, memory_order_release);
    ha->repl_ack_count = 0;        /* reset ack counter for this round */
    /* Push config and replicate WAL to standbys (Req 7.1, 7.2) */
    push_config_to_standbys(c, ...);
    ha_send_ctrlhello_to_all_standbys(c);   /* triggers catch-up replication */
    /* Arm committing timeout (degraded-mode fallback, Req 8.4) */
    ha_arm_committing_timeout(ha, c);
} else {
    if (is_first_boot) {
        cluster_init_phase2(c);    /* init structures; config arrives via primary push */
        ha->phase2_done = 1;
    }
    atomic_store(&ha->role, CTRL_STANDBY, memory_order_release);
    /* re-arm: heartbeat watchdog fires ha_on_primary_ctrl_fd_rdhup on next timeout */
}

ha_repl_ack_received() — called when a standby acks config+WAL catch-up

(Note: this function does not exist in current code — superseded by ha-single-leader-role, see that spec's design.md. Standby ack counting toward promotion was removed along with CTRL_PRIMARY_COMMITTING; job-traffic admission is now ha_has_write_quorum()'s live count of connected standbys, evaluated per-request inside ha_dispatch_gate, not a one-time per-round tally.)

void ha_repl_ack_received(ha_state_t *ha, cluster_t *c) {
    int acks = atomic_fetch_add(&ha->repl_ack_count, 1) + 1;
    int quorum = (ha->n_peers + 2) / 2;  /* majority of total cluster */
    if (acks >= quorum - 1) {             /* -1 because primary counts itself */
        ha_disarm_committing_timeout(ha);
        ha_promote_to_primary(c);         /* sets role=CTRL_PRIMARY, enables hooks */
    }
}

Called from the existing handle_ctrlhello reply path (CtrlHelloReply ack from standby after catch-up) and from push_config_to_standbys ack handler.

handle_request_vote role gate (extended)

(Note: superseded by raft-adapter-conformance, see that spec's design.md. handle_request_vote is now #ifdef XCE_TESTING-only, kept as a test harness entry point; production vote-request dispatch is cluster.c's case OPCode_requestVote:, which applies its own freshness gate and then calls raft_on_vote_request() — the PRIMARY-role deny is now on_vote_eligible/RAFT_LEADER inside the raft library, not this snippet. CTRL_PRIMARY_COMMITTING in the snippet below also no longer exists — see the Key Decisions note above.)

ctrl_role_t my_role = atomic_load_explicit(&ha->role, memory_order_acquire);
if (my_role == CTRL_PRIMARY || my_role == CTRL_PRIMARY_COMMITTING)
    return 0;  /* deny — I am primary (or committing); don't grant votes to challengers */
if (my_role == CTRL_STANDBY && ha_heartbeat_is_fresh(ha))
    return 0;  /* deny — my primary is alive */
/* CTRL_INIT, CTRL_ELECTING, or STANDBY without live primary: evaluate Raft gates below */

ha_heartbeat_is_fresh()

(Fixed: signature and field name were stale — corrected to match current ha.c. The function is non-static (exported) and reads ha->last_activity, a whole-second time_t, not a struct timespec last_heartbeat — the design's original nsec-precision sketch was implemented as whole-second precision, and it now null-guards last_activity == 0 for "no primary message seen yet".)

int ha_heartbeat_is_fresh(ha_state_t *ha) {
    time_t last = ha->last_activity;   /* time_t load is atomic on LP64 */
    if (last == 0)
        return 0; /* no primary message seen yet */
    struct timespec now;
    clock_gettime(CLOCK_MONOTONIC, &now);
    long ms = (long)(now.tv_sec - last) * 1000L;
    return ms < (long)ha->config.heartbeat_timeout_ms;
}

myapp_run (restructured — no blocking)

int myapp_run(myconf_t *myconf) {
    cluster_t c; memset(&c, 0, sizeof(c)); c.myconf = myconf;
    cluster_init_phase1(&c);              /* tunables, managers, endpoints zarray */
    initialize_openssl();
    c.ha.ssl_ctx        = create_ssl_context(myconf->certfile, myconf->keyfile);
    c.ha.client_ssl_ctx = SSL_CTX_new(TLS_client_method());
    SSL_CTX_set_verify(c.ha.client_ssl_ctx, SSL_VERIFY_NONE, NULL);
    iomgr_init(&c.iomgr); iomgr_set_ssl_ctx(&c.iomgr, c.ha.ssl_ctx);
    vtaskmgr_init(&c.vtaskmgr, nvcpus);
    iomgr_start(&c.iomgr); vtaskmgr_start(&c.vtaskmgr);
    cluster_register_listener(&c);        /* listener in epoll before election */
    delaymgr_init(&c.delaymgr); delaymgr_start(&c.delaymgr);  /* moved from cluster_start */
    persist_init(&c.persist, ...); persist_start(&c.persist);   /* moved from cluster_start */
    load_primary_config(&c);              /* pre-load config_tag for freshness gate */
    /* bootstrap_config_if_needed() REMOVED — freshness gate + primary push replaces it */
    /* Build peer array from myconf->peers_cfg ... */
    ha_start_election_async(&c.ha, &c, n_peers, peer_ptrs, my_tag, my_lsn, myconf->name);
    return cluster_run(&c);               /* unchanged — just waits for stop signal */
}

Note: vote_ep_ctx_t carries cluster_t *c so that ha_elect_evaluate_and_signal (called from vtask context via vote_reply_msg_handler§) has access to cluster_t for cluster_init_phase2, push_config_to_standbys, and ha_send_ctrlhello_to_all_standbys. § (renamed/generalized to peer_ctrl_msg_handler — superseded by ha-prevote.)

ha_dispatch_gate — CTRL_PRIMARY_COMMITTING case

(Note: superseded by ha-single-leader-role, see that spec's design.md. This case does not exist — ha_dispatch_gate's switch has only CTRL_PRIMARY, CTRL_INIT, CTRL_ELECTING, CTRL_STANDBY. The write-admission gating this case describes now lives inside the CTRL_PRIMARY case itself, via ha_has_write_quorum(): state-changing ops are refused with XCE_ERR_NOT_READY when the live connected-standby count is below majority, while the same replication/read-only opcode allowlist below — plus fact-bearing ops (OPCode_notify/update/setpreason/clrpreason) — stays admitted.)

case CTRL_PRIMARY_COMMITTING:
    /* Allow replication acks (config push ack, CtrlHelloReply, WAL ack) so
     * ha_repl_ack_received() can count them toward majority.
     * Deny job dispatch and queue/resource management until majority ack. */
    switch (opcode) {
    case OPCode_heartbeat:
    case OPCode_replConfig:        /* config push ack from standby */
    case OPCode_replConfigPush:
    case OPCode_upgradeConfigAck:
    case OPCode_hello:             /* CtrlHelloReply ack from standby after catch-up */
    case OPCode_walAck:
    case OPCode_replEntry:
    case OPCode_replAck:
    case OPCode_replCatchUpDone:   /* standby signals catch-up complete → count ack */
        return 0;
    default:
        return -XCE_ERR_NOT_READY;
    }

OPCode_replCatchUpDone handler in cluster.c calls ha_repl_ack_received(ha, c) to increment the ack counter and conditionally transition to CTRL_PRIMARY. (Note: ha_repl_ack_received does not exist — see notes above; superseded by ha-single-leader-role.)

Error Handling

• Connect failure: endpoint skipped; peer absent from election. After watchdog timeout, election evaluates with available votes only.
• SSL handshake failure: vote_ep_err_handler calls handle_request_vote_reply(grant=0).
• cluster_init_phase2 failure from vtask: logs ERROR; posts stop to wbuf → clean shutdown.
• Client during CTRL_ELECTING or CTRL_PRIMARY_COMMITTING: ha_dispatch_gate returns XCE_ERR_NOT_READY; client retries. (Note: CTRL_PRIMARY_COMMITTING no longer exists — superseded by ha-single-leader-role; the equivalent current behavior is ha_dispatch_gate's CTRL_PRIMARY case refusing state-changing ops via ha_has_write_quorum() until majority standbys are connected.)
• Committing timeout (Req 8.4): if majority ack not received within committing_timeout_ms, watchdog fires ha_promote_to_primary anyway (degraded mode) with a WARN log. Prevents deadlock when standbys are unavailable at startup.
• Vote re-arm on STANDBY: ha_elect_evaluate_and_signal re-arms ha_start_election_async. Heartbeat watchdog fires ha_on_primary_ctrl_fd_rdhup → CTRL_ELECTING → re-election.

Testing Strategy

Unit Tests

1. ha_start_election_async returns in < 5 ms without blocking
2. on_vote_connected encodes and sends correct OP_requestVote
3. vote_reply_msg_handler reads capnp fields before free — ASAN clean
4. handle_request_vote role gate: PRIMARY→deny; STANDBY+fresh→deny; STANDBY+stale→evaluate; ELECTING→evaluate
5. ha_elect_evaluate_and_signal enters COMMITTING then PRIMARY after majority ack
6. Election watchdog fires within election_timeout_ms + 200ms when no replies
7. ha_repl_ack_received: majority acks → CTRL_PRIMARY; below majority → stays COMMITTING
8. Committing timeout watchdog: promotes to PRIMARY (degraded) if acks don't arrive

Integration Tests

9. 3-node startup: ctrl-a wins election → CTRL_PRIMARY_COMMITTING → pushes config to ctrl-b/c → receives replCatchUpDone × 2 → CTRL_PRIMARY → serves jobs
10. ctrl-b/c start with config_tag=0; receive config via primary push (no independent fetch); confirm via replCatchUpDone
11. Primary crash (SIGKILL): ctrl-b heartbeat timeout → re-election → COMMITTING → ack from ctrl-c → PRIMARY; serves traffic from last WAL state
12. Client during COMMITTING receives XCE_ERR_NOT_READY and retries successfully after gate opens

Regression Tests

13. Single-node self-elects; no standbys → committing timeout fires → degraded PRIMARY immediately
14. ASAN clean: no UAF, no stack-use-after-return in election or committing path