myxce/.kiro/specs/election-iomgr-refactor/requirements.md

Requirements Document

Introduction

The XCE controller operates a 3-node HA cluster where one node is elected PRIMARY before serving jobs. The current election mechanism mixes blocking per-peer pthreads with the iomgr epoll event loop, causing SSL context confusion, capnp use-after-free, and race conditions.

This feature replaces the blocking pthread-based election with a fully async iomgr-driven implementation, and formalizes the complete HA protocol: election by config freshness, replication push after primary determined, and availability gate after majority ack.

Boundary Context

• In scope: Outgoing vote RPC transport; SSL context lifecycle for vote connections; election quorum; role-based vote grant/deny; config+state replication push to standbys after election; availability gate (majority ack before serving traffic).
• Out of scope: Incoming vote RPC handling (already correct in cluster.c); dynamic cluster membership changes (candidate list is static); config schema changes.
• Adjacent expectations: iomgr started and listener registered before election begins. The FSM state machine and ha_dispatch_gate remain the authority on opcode routing.

Requirements

1. No Blocking Threads for Outgoing Vote RPCs

Objective: As the cluster operator, I want the controller election to complete reliably without SSL errors or memory corruption, so that the cluster reaches PRIMARY state on every startup.

Acceptance Criteria

1. The controller shall not create pthreads whose sole purpose is to send a vote request and block waiting for a reply during election.
2. When the controller initiates an election round, the controller shall send OP_requestVote to each peer using the same I/O mechanism used for all other outgoing RPCs.
3. If a peer does not respond within the election probe timeout, the controller shall treat that peer as a denied vote without stalling the event loop.

---

2. Async Outgoing Vote Connection via iomgr

Objective: As the controller process, I want outgoing vote connections managed by the iomgr event loop, so that SSL handshake and data transfer do not block other I/O.

Acceptance Criteria

1. When the controller sends a vote request to a peer, the controller shall establish the TCP connection and complete the TLS handshake without blocking the main event loop thread.
2. The controller shall use a TLS client context (not a server context) for outgoing vote connections.
3. When the TLS handshake with a peer completes, the controller shall send the OP_requestVote packet through the established connection.
4. If the TCP connection or TLS handshake to a peer fails, the controller shall count that peer as a denied vote and continue the election with the remaining peers.

---

3. Vote Reply Handling Through iomgr Dispatch

Objective: As the controller process, I want vote replies to arrive as normal iomgr packets, so that the capnp segment lifetime is managed consistently with all other RPCs.

Acceptance Criteria

1. When a peer sends a vote reply, the controller shall decode it through the standard iomgr packet-receive path rather than a raw blocking read.
2. The controller shall not access capnp-decoded fields after the capnp segment for that packet has been freed.
3. When a vote reply is received, the controller shall extract the peer's term, vote_granted, peer_config_tag, and peer_last_log_lsn and update the election tally.
4. If the peer's term in a vote reply is greater than the candidate's current term, the controller shall update its own term before recording the vote result.

---

4. Election Quorum, Ground-Truth, Timeout, and Retry

Objective: As the cluster operator, I want the election to converge to a single PRIMARY that holds the ground-truth cluster state, so that no data is lost and the cluster becomes available even if nodes start at different times.

Acceptance Criteria

1. When vote replies from a majority of candidate nodes (≥ ⌈(n+1)/2⌉) grant the vote, the controller shall transition to CTRL_PRIMARY_COMMITTING (pre-availability state).
2. The controller shall grant a vote to a candidate only if the candidate's config_tag is greater than or equal to the voter's own config_tag. This freshness gate guarantees that the elected primary holds the ground-truth configuration — no post-election config fetch is ever needed by the winner.
3. On cold start, only the node holding the highest config_tag can win election. All other nodes (config_tag=0 on first boot) will lose the freshness gate and cannot become primary.
4. When all peers have replied but no majority granted the vote, the controller shall retry after the next heartbeat timeout.
5. When no vote replies arrive within the election probe timeout, the controller shall complete the round as a split vote and retry.
6. While in CTRL_ELECTING state, if a heartbeat is received from a node that has become PRIMARY, the controller shall transition to CTRL_STANDBY and abandon the election.
7. The controller shall retry elections indefinitely until it wins a quorum or detects an existing primary via heartbeat.

---

5. FSM State Consistency

Objective: As the controller process, I want the FSM state to accurately reflect the controller's role, so that the dispatch gate correctly accepts or rejects incoming RPCs.

Acceptance Criteria

1. While in CTRL_ELECTING state, the controller shall accept OP_requestVote and OP_probeConfigTag and shall reply with XCE_ERR_NOT_READY to all other opcodes.
2. While in CTRL_PRIMARY_COMMITTING state, the controller shall accept replication-related opcodes (config push ack, WAL ack) and shall reply with XCE_ERR_NOT_READY to job dispatch and queue management opcodes.
3. When the controller transitions to CTRL_PRIMARY (fully available), it shall begin accepting job dispatch and queue management opcodes.
4. When the controller transitions to CTRL_STANDBY, it shall begin accepting replication and heartbeat opcodes.
5. The controller shall maintain a monotonically increasing term counter across election rounds and persist it to disk so the term survives a process restart.

---

6. Vote-Granting Rules Based on Voter Role

Objective: As the cluster operator, I want votes granted only when the voter has no live primary, so that a healthy primary is never displaced, and a crashed primary triggers a new election only after the heartbeat timeout expires.

Acceptance Criteria

1. While in CTRL_PRIMARY state, the controller shall deny all incoming OP_requestVote requests regardless of the candidate's term or config_tag.
2. While in CTRL_STANDBY state with a live primary connection (heartbeat received within heartbeat_timeout_ms), the controller shall deny all incoming OP_requestVote requests.
3. When in CTRL_STANDBY state and the primary heartbeat has not been received within heartbeat_timeout_ms, the controller shall transition to CTRL_ELECTING and begin granting votes to eligible candidates.
4. While in CTRL_INIT or CTRL_ELECTING state, the controller shall evaluate incoming OP_requestVote requests using the term, voted_for, and freshness gates (config_tag, last_log_lsn) without any role-based denial.
5. When a primary crashes (stops sending heartbeats), the surviving standbys shall detect the loss within heartbeat_timeout_ms, transition to CTRL_ELECTING, and elect a new primary from among themselves using the last committed state.

---

7. Post-Election Config and State Replication

Objective: As the cluster operator, I want the elected primary to immediately propagate its ground-truth configuration and cluster state to all standbys, so that standbys have up-to-date state and can become primary on failover without any post-election config fetch.

Acceptance Criteria

1. When the controller wins election and enters CTRL_PRIMARY_COMMITTING, it shall push the cluster configuration to all peer controllers that have a lower config_tag.
2. When the controller wins election, it shall replicate its WAL (committed job/queue/worker state) to all standbys via the existing catch-up replication mechanism.
3. When a new standby starts with config_tag=0 and loses the election, it shall receive the cluster configuration via the primary's replication push — it shall NOT attempt an independent config fetch before the primary pushes it.
4. The controller shall confirm config and state receipt from each standby via acknowledgement before counting that standby toward the majority ack threshold.

---

8. Primary Availability Gate

Objective: As the cluster operator, I want the primary to begin serving job traffic only after the majority of controllers have confirmed they hold the replicated state, so that failover can always promote a standby with current data.

Acceptance Criteria

1. After winning election, the controller shall enter CTRL_PRIMARY_COMMITTING and shall not accept job dispatch or queue management requests until the availability gate opens.
2. When a majority of peer controllers (≥ ⌈(n+1)/2⌉ − 1 peers, since primary counts itself) have acknowledged receipt of the current config and committed WAL state, the controller shall transition from CTRL_PRIMARY_COMMITTING to CTRL_PRIMARY.
3. When CTRL_PRIMARY state is reached, the controller shall begin accepting all client opcodes including job dispatch, queue control, and resource management.
4. If the majority ack threshold cannot be reached within a configurable timeout, the controller shall log a warning and open anyway (degraded mode) so the cluster does not deadlock when standbys are unavailable at startup.

---

9. No Regression in Existing Behaviors

Objective: As the cluster operator, I want all existing behaviors preserved after the refactor, so that nodes that previously worked correctly continue to do so.

Acceptance Criteria

1. When a single-node cluster starts (candidate count = 1), the controller shall self-elect to CTRL_PRIMARY without contacting any peers.
2. The elected primary shall never need to fetch config from another node — the vote freshness gate guarantees it already holds the highest config_tag.
3. When the primary's control channel connection drops (RDHUP), the controller shall trigger a new election. The winning standby shall promote via ha_promote_to_primary (hooks only — its state is already current via continuous replication).
4. The controller shall prioritize election by node priority index: lower-index nodes start campaigning first, reducing split votes on simultaneous startup.