Rongsong Shen
e31673dd88
Captures all uncommitted work from ha-config-bootstrap, ha-heartbeat-liveness, and associated spec/tooling updates: - src: ha.c, ha.h, cluster.c, config_repl.c, myapp.c, raft.c — heartbeat liveness, RDHUP integration, committing phase fixes - tests: test_ha_role.c, test_config_repl.c, test_ha_rdhup_integration.c, test_p1_integration.c, test_p3_hardening.c, scheduler test - specs: election-iomgr-refactor, ha-heartbeat-liveness, ha-peer-connection-reuse, raft-library (new); ha-config-bootstrap and election-iomgr-refactor task/research updates - tooling: .wolf/ state, ha/data cluster fixtures, raft/raft.tla, src/build symlink, xce-build skill update
206 lines
10 KiB
Markdown
206 lines
10 KiB
Markdown
# Requirements Document
|
||
|
||
## Introduction
|
||
|
||
The XCE controller operates a 3-node HA cluster where one node is elected PRIMARY before
|
||
serving jobs. The current election mechanism mixes blocking per-peer pthreads with the iomgr
|
||
epoll event loop, causing SSL context confusion, capnp use-after-free, and race conditions.
|
||
|
||
This feature replaces the blocking pthread-based election with a fully async iomgr-driven
|
||
implementation, and formalizes the complete HA protocol: election by config freshness,
|
||
replication push after primary determined, and availability gate after majority ack.
|
||
|
||
## Boundary Context
|
||
|
||
- **In scope**: Outgoing vote RPC transport; SSL context lifecycle for vote connections;
|
||
election quorum; role-based vote grant/deny; config+state replication push to standbys
|
||
after election; availability gate (majority ack before serving traffic).
|
||
- **Out of scope**: Incoming vote RPC handling (already correct in cluster.c);
|
||
dynamic cluster membership changes (candidate list is static); config schema changes.
|
||
- **Adjacent expectations**: iomgr started and listener registered before election begins.
|
||
The FSM state machine and `ha_dispatch_gate` remain the authority on opcode routing.
|
||
|
||
## Requirements
|
||
|
||
### 1. No Blocking Threads for Outgoing Vote RPCs
|
||
|
||
**Objective:** As the cluster operator, I want the controller election to complete reliably
|
||
without SSL errors or memory corruption, so that the cluster reaches PRIMARY state on every
|
||
startup.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. The controller shall not create pthreads whose sole purpose is to send a vote request and
|
||
block waiting for a reply during election.
|
||
2. When the controller initiates an election round, the controller shall send `OP_requestVote`
|
||
to each peer using the same I/O mechanism used for all other outgoing RPCs.
|
||
3. If a peer does not respond within the election probe timeout, the controller shall treat
|
||
that peer as a denied vote without stalling the event loop.
|
||
|
||
---
|
||
|
||
### 2. Async Outgoing Vote Connection via iomgr
|
||
|
||
**Objective:** As the controller process, I want outgoing vote connections managed by the
|
||
iomgr event loop, so that SSL handshake and data transfer do not block other I/O.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. When the controller sends a vote request to a peer, the controller shall establish the TCP
|
||
connection and complete the TLS handshake without blocking the main event loop thread.
|
||
2. The controller shall use a TLS client context (not a server context) for outgoing vote
|
||
connections.
|
||
3. When the TLS handshake with a peer completes, the controller shall send the
|
||
`OP_requestVote` packet through the established connection.
|
||
4. If the TCP connection or TLS handshake to a peer fails, the controller shall count that
|
||
peer as a denied vote and continue the election with the remaining peers.
|
||
|
||
---
|
||
|
||
### 3. Vote Reply Handling Through iomgr Dispatch
|
||
|
||
**Objective:** As the controller process, I want vote replies to arrive as normal iomgr
|
||
packets, so that the capnp segment lifetime is managed consistently with all other RPCs.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. When a peer sends a vote reply, the controller shall decode it through the standard iomgr
|
||
packet-receive path rather than a raw blocking `read`.
|
||
2. The controller shall not access capnp-decoded fields after the capnp segment for that
|
||
packet has been freed.
|
||
3. When a vote reply is received, the controller shall extract the peer's `term`,
|
||
`vote_granted`, `peer_config_tag`, and `peer_last_log_lsn` and update the election tally.
|
||
4. If the peer's term in a vote reply is greater than the candidate's current term, the
|
||
controller shall update its own term before recording the vote result.
|
||
|
||
---
|
||
|
||
### 4. Election Quorum, Ground-Truth, Timeout, and Retry
|
||
|
||
**Objective:** As the cluster operator, I want the election to converge to a single PRIMARY
|
||
that holds the ground-truth cluster state, so that no data is lost and the cluster becomes
|
||
available even if nodes start at different times.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. When vote replies from a majority of candidate nodes (≥ ⌈(n+1)/2⌉) grant the vote, the
|
||
controller shall transition to CTRL_PRIMARY_COMMITTING (pre-availability state).
|
||
2. The controller shall grant a vote to a candidate only if the candidate's `config_tag` is
|
||
greater than or equal to the voter's own `config_tag`. This freshness gate guarantees that
|
||
the elected primary holds the ground-truth configuration — no post-election config fetch
|
||
is ever needed by the winner.
|
||
3. On cold start, only the node holding the highest `config_tag` can win election. All other
|
||
nodes (config_tag=0 on first boot) will lose the freshness gate and cannot become primary.
|
||
4. When all peers have replied but no majority granted the vote, the controller shall retry
|
||
after the next heartbeat timeout.
|
||
5. When no vote replies arrive within the election probe timeout, the controller shall
|
||
complete the round as a split vote and retry.
|
||
6. While in CTRL_ELECTING state, if a heartbeat is received from a node that has become
|
||
PRIMARY, the controller shall transition to CTRL_STANDBY and abandon the election.
|
||
7. The controller shall retry elections indefinitely until it wins a quorum or detects an
|
||
existing primary via heartbeat.
|
||
|
||
---
|
||
|
||
### 5. FSM State Consistency
|
||
|
||
**Objective:** As the controller process, I want the FSM state to accurately reflect the
|
||
controller's role, so that the dispatch gate correctly accepts or rejects incoming RPCs.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. While in CTRL_ELECTING state, the controller shall accept `OP_requestVote` and
|
||
`OP_probeConfigTag` and shall reply with `XCE_ERR_NOT_READY` to all other opcodes.
|
||
2. While in CTRL_PRIMARY_COMMITTING state, the controller shall accept replication-related
|
||
opcodes (config push ack, WAL ack) and shall reply with `XCE_ERR_NOT_READY` to job
|
||
dispatch and queue management opcodes.
|
||
3. When the controller transitions to CTRL_PRIMARY (fully available), it shall begin accepting
|
||
job dispatch and queue management opcodes.
|
||
4. When the controller transitions to CTRL_STANDBY, it shall begin accepting replication and
|
||
heartbeat opcodes.
|
||
5. The controller shall maintain a monotonically increasing term counter across election rounds
|
||
and persist it to disk so the term survives a process restart.
|
||
|
||
---
|
||
|
||
### 6. Vote-Granting Rules Based on Voter Role
|
||
|
||
**Objective:** As the cluster operator, I want votes granted only when the voter has no live
|
||
primary, so that a healthy primary is never displaced, and a crashed primary triggers a new
|
||
election only after the heartbeat timeout expires.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. While in CTRL_PRIMARY state, the controller shall deny all incoming `OP_requestVote`
|
||
requests regardless of the candidate's term or config_tag.
|
||
2. While in CTRL_STANDBY state with a live primary connection (heartbeat received within
|
||
`heartbeat_timeout_ms`), the controller shall deny all incoming `OP_requestVote` requests.
|
||
3. When in CTRL_STANDBY state and the primary heartbeat has not been received within
|
||
`heartbeat_timeout_ms`, the controller shall transition to CTRL_ELECTING and begin
|
||
granting votes to eligible candidates.
|
||
4. While in CTRL_INIT or CTRL_ELECTING state, the controller shall evaluate incoming
|
||
`OP_requestVote` requests using the term, voted_for, and freshness gates (config_tag,
|
||
last_log_lsn) without any role-based denial.
|
||
5. When a primary crashes (stops sending heartbeats), the surviving standbys shall detect
|
||
the loss within `heartbeat_timeout_ms`, transition to CTRL_ELECTING, and elect a new
|
||
primary from among themselves using the last committed state.
|
||
|
||
---
|
||
|
||
### 7. Post-Election Config and State Replication
|
||
|
||
**Objective:** As the cluster operator, I want the elected primary to immediately propagate
|
||
its ground-truth configuration and cluster state to all standbys, so that standbys have
|
||
up-to-date state and can become primary on failover without any post-election config fetch.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. When the controller wins election and enters CTRL_PRIMARY_COMMITTING, it shall push the
|
||
cluster configuration to all peer controllers that have a lower `config_tag`.
|
||
2. When the controller wins election, it shall replicate its WAL (committed job/queue/worker
|
||
state) to all standbys via the existing catch-up replication mechanism.
|
||
3. When a new standby starts with `config_tag=0` and loses the election, it shall receive the
|
||
cluster configuration via the primary's replication push — it shall NOT attempt an
|
||
independent config fetch before the primary pushes it.
|
||
4. The controller shall confirm config and state receipt from each standby via acknowledgement
|
||
before counting that standby toward the majority ack threshold.
|
||
|
||
---
|
||
|
||
### 8. Primary Availability Gate
|
||
|
||
**Objective:** As the cluster operator, I want the primary to begin serving job traffic only
|
||
after the majority of controllers have confirmed they hold the replicated state, so that
|
||
failover can always promote a standby with current data.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. After winning election, the controller shall enter CTRL_PRIMARY_COMMITTING and shall not
|
||
accept job dispatch or queue management requests until the availability gate opens.
|
||
2. When a majority of peer controllers (≥ ⌈(n+1)/2⌉ − 1 peers, since primary counts itself)
|
||
have acknowledged receipt of the current config and committed WAL state, the controller
|
||
shall transition from CTRL_PRIMARY_COMMITTING to CTRL_PRIMARY.
|
||
3. When CTRL_PRIMARY state is reached, the controller shall begin accepting all client
|
||
opcodes including job dispatch, queue control, and resource management.
|
||
4. If the majority ack threshold cannot be reached within a configurable timeout, the
|
||
controller shall log a warning and open anyway (degraded mode) so the cluster does not
|
||
deadlock when standbys are unavailable at startup.
|
||
|
||
---
|
||
|
||
### 9. No Regression in Existing Behaviors
|
||
|
||
**Objective:** As the cluster operator, I want all existing behaviors preserved after the
|
||
refactor, so that nodes that previously worked correctly continue to do so.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. When a single-node cluster starts (candidate count = 1), the controller shall self-elect
|
||
to CTRL_PRIMARY without contacting any peers.
|
||
2. The elected primary shall never need to fetch config from another node — the vote
|
||
freshness gate guarantees it already holds the highest config_tag.
|
||
3. When the primary's control channel connection drops (RDHUP), the controller shall trigger a
|
||
new election. The winning standby shall promote via `ha_promote_to_primary` (hooks only —
|
||
its state is already current via continuous replication).
|
||
4. The controller shall prioritize election by node priority index: lower-index nodes start
|
||
campaigning first, reducing split votes on simultaneous startup.
|