myxce/.kiro/specs/raft-drive-async/requirements.md

Requirements Document

Introduction

The XCE controller's Raft protocol drive is not fully event-driven: blocking operations run inline on the two threads that carry cluster liveness signals — the iomgr epoll loop and the delaymgr timer thread. A blocking audit (2026-06-25) found three CRITICAL violations of the corrected ha-async-io.md thread-role contract: snapshot send + checkpoint inline on iomgr (ha.c:2702, config_repl.c:2446), config-push busy-poll + join inline on iomgr (config_repl.c:1332), and blocking gethostbyname+connect on the delaymgr heartbeat thread (netio.c:289/302). Root cause: peer_ctrl_msg_handler is registered directly as the iomgr message handler (ha.c:4265/4307/6349) instead of dispatching to the vtask pool as normal cluster messages do (cluster.c:711). When a drive thread stalls, heartbeats/votes/ACKs stop flowing, peers declare the leader dead, and a spurious election follows — the likely root of the bug-851 chaotic-failover family.

This feature brings the controller into compliance with the steering thread-role contract (ha-async-io.md, corrected 2026-06-25) so that no liveness-carrying thread blocks inline, while preserving every existing HA behavior and the persist-before-ACK / synchronous-commit policies. It is verified by a new network-partition (peer unreachable, not merely killed) regression scenario that the current process-kill-only soak cannot reproduce.

The authoritative subject for the acceptance criteria below is the XCE controller and its named drive threads (iomgr loop, delaymgr timer, vtask pool).

Boundary Context

• In scope: keeping the iomgr loop and delaymgr timer non-blocking; moving long replication/config I/O (snapshot transfer, config-push) off the drive threads to an admission-controlled execution path; bounding and offloading the peer connect; preserving per-connection message ordering and processing correctness; admission protection so liveness-critical message handling is never starved; a network-partition regression test; no regression of existing HA features.
• Out of scope: the commit-barrier async conversion (repl_wait_majority_ack → event-driven resolve) — owned by raft-apply-decoupling as prerequisite P1; the apply-after-commit / hook-decoupling refactor; asynchronous DNS resolution (deferred unless measured — HPC peers use IP//etc/hosts); the snapshot content/format (entity-bearing snapshots is a separate spec) — this feature only moves the snapshot transfer off the drive threads; the MEDIUM vtask-pool blockers beyond what admission control covers.
• Adjacent expectations: relies on the corrected ha-async-io.md thread-role contract as the authority; reuses the existing COMMIT_WAIT_RESERVED admission cap and the xce_probe_config_tag bounded-connect pattern; consumes election-iomgr-refactor (completed) as upstream for its primitives (delay_task_ms, on_connected, non-blocking vote fan-out) and stays a standalone spec (scope-check 2026-06-25: ADJACENT-SAME-AREA — that spec fixed startup blocking, this fixes runtime drive-thread blocking); touches the same peer_ctrl_msg_handler/repl-dispatch region as ha-term-safety; preserves the durability and synchronous-commit guarantees owned by ha-commit-durability and ha-commit-engine.

Requirements

Requirement 1: Liveness During Long-Running Replication and Configuration I/O

Objective: As a cluster operator, I want the controller to keep heartbeating and processing protocol messages while it transfers a snapshot or pushes a configuration to a standby, so that a large or slow transfer to one peer does not trigger a false failover of the cluster.

Acceptance Criteria

1. While the controller is transferring a snapshot to one standby, the controller shall continue to send heartbeats to every other peer within the configured heartbeat interval (default 3000 ms).
2. While the controller is pushing a configuration to standbys, the controller shall continue to send heartbeats to every peer within the configured heartbeat interval and continue to process inbound votes and acknowledgments.
3. When the controller initiates a snapshot transfer, checkpoint creation, or configuration push, the iomgr loop shall remain available to dispatch other inbound messages without waiting for that operation to complete.
4. While a long-running replication or configuration operation is executing, the controller shall continue to process inbound RequestVote messages and respond within the configured election timeout (default 5000 ms).
5. If a snapshot transfer to one standby is already in progress, then the controller shall not start a second concurrent transfer to that same standby.

Requirement 2: Liveness and Bounded Behavior When a Peer Is Unreachable

Objective: As a cluster operator, I want an unreachable peer to never stall heartbeats to the healthy peers, so that one partitioned node cannot cause the surviving majority to elect a new leader unnecessarily.

Acceptance Criteria

1. While one peer is unreachable (its host accepts no connection or silently drops traffic), the controller shall continue to send heartbeats to all reachable peers within the configured heartbeat interval.
2. While one peer is unreachable, the delaymgr timer thread shall continue to fire periodic and election-timeout callbacks without being delayed by the connection attempt to the unreachable peer.
3. When the controller attempts to connect to a peer, the controller shall bound the connection attempt to a configured timeout and report failure on expiry rather than waiting for the operating-system connect timeout.
4. If a connection attempt to a peer fails or times out, then the controller shall retry the connection later without terminating, consistent with the "controllers never self-terminate" invariant.
5. When the controller is already attempting to (re)connect to a given peer, the controller shall not enqueue a duplicate concurrent connection attempt for that same peer.
6. When a connection attempt to a peer succeeds, the controller shall register and begin using the new connection without losing or reordering messages already queued for that peer.

Requirement 3: Worker-Pool Admission Protection

Objective: As a cluster operator, I want offloaded blocking work to never consume all worker capacity, so that liveness-critical message handling always has a worker available even under replication or reconnection load.

Acceptance Criteria

1. The controller shall reserve worker capacity such that liveness-critical message handling (heartbeats, votes, acknowledgments) always has at least one available worker even when blocking offloaded tasks are in flight.
2. While the maximum number of admitted blocking offloaded tasks is in flight, the controller shall continue to admit and process non-blocking liveness-critical messages.
3. When the number of concurrently requested blocking offloaded tasks exceeds the admission limit, the controller shall queue or defer the excess rather than block a liveness-carrying thread.
4. The controller shall route long/unbounded blocking I/O (snapshot transfer, configuration push, peer reconnect) to an execution path that is subject to the admission limit, not to the iomgr loop or the delaymgr timer thread.

Requirement 4: Message Ordering and Processing Correctness Preserved

Objective: As a cluster operator, I want peer control messages to be processed with the same ordering and correctness as today after they are moved off the iomgr loop, so that consensus correctness is unaffected by the threading change.

Acceptance Criteria

1. When peer control messages arrive on a single connection, the controller shall handle (process/apply) them in arrival order, even though handling is dispatched to the worker pool rather than run inline on the iomgr loop. (This concerns inbound processing order, distinct from the already-guaranteed outbound send order of the endpoint output queue.)
2. While handling a peer control message for a given connection, the controller shall not begin handling another message from that same connection until the first completes, so per-connection processing remains strictly sequential.
3. While handling a peer control message, the controller shall not hold its connection's sequential-processing exclusivity across a long-running offloaded operation (snapshot transfer, configuration push); such operations shall be triggered and the handler shall return, so the connection's subsequent messages (including its heartbeat acknowledgments) are not queued behind the long operation.
4. When peer control message processing mutates iomgr-owned connection or epoll state, the controller shall do so only through the thread-safe (lock-guarded) registration interface, without data races against the iomgr event loop.
5. The controller shall preserve the existing semantics of vote handling, acknowledgment handling, and term updates after relocating peer control message processing off the iomgr loop.
6. The controller shall continue to persist term and vote state before responding to a vote, preserving the existing durability ordering on the vote path.

Requirement 5: Thread-Role Contract Compliance

Objective: As a maintainer, I want the controller's drive threads to obey the steering thread-role contract by construction, so that future changes cannot reintroduce inline blocking on a liveness thread.

Acceptance Criteria

1. The iomgr loop shall perform only non-blocking I/O and message dispatch; it shall not execute blocking calls or inline message processing.
2. The delaymgr timer thread shall only fire timers and trigger work; it shall not execute blocking calls (connect, file I/O, or busy-wait sleeps).
3. The controller shall execute bounded blocking work in the worker pool only under the admission reservation, and long/unbounded blocking I/O only on the dedicated offload path.
4. The controller shall conform to the thread-role contract defined in ha-async-io.md for all drive-thread code paths brought into scope by this feature.

Requirement 6: No Regression of Existing HA Features

Objective: As a cluster operator, I want all existing HA behaviors to keep working unchanged after the drive is made non-blocking, so that the hardening introduces no functional regression.

Acceptance Criteria

1. When the existing controller test suites for election/voting, heartbeat/liveness, failover, snapshot recovery, configuration push/upgrade, WAL replication/catch-up, and the commit barrier are run, the controller shall pass them without modification to their asserted behavior.
2. The controller shall continue to elect a single primary on quorum, increment terms, and reject stale-term votes exactly as the existing election tests assert.
3. The controller shall continue to detect a dead primary and fail over within the configured heartbeat timeout (default 10000 ms) exactly as the existing liveness/failover tests assert.
4. The controller shall continue to deliver snapshots and replicated WAL entries to standbys and recover them on restart exactly as the existing replication and recovery tests assert.
5. The controller shall continue to push and apply configuration changes through majority acknowledgment exactly as the existing config-replication tests assert.
6. The controller shall preserve the persist-before-ACK durability guarantee and the synchronous commit-barrier behavior, which this feature does not modify.
7. If any existing HA regression test fails after the change, then the feature shall be considered incomplete until the behavior is restored.

Requirement 7: Network-Partition Verification

Objective: As a cluster operator, I want a regression test that makes a peer unreachable (not merely killed), so that the heartbeat-starvation failure mode this feature fixes is provably closed and stays closed.

Acceptance Criteria

1. The verification suite shall include a scenario that renders one peer unreachable by dropping its traffic (network partition / blackhole), distinct from process termination.
2. When one peer is made unreachable during the scenario, the verification shall assert that heartbeats to the surviving peers continue and that no spurious election of a new leader occurs while the existing leader remains alive and reachable to the majority.
3. When the partition is healed, the verification shall assert that the previously unreachable peer rejoins and catches up without operator intervention.
4. While the partition scenario runs, the verification shall assert that no committed job is lost, consistent with the existing failover soak guarantees.

Requirement 8: Heartbeat Fast-Path Performance

Objective: As a cluster operator, I want the steady-state heartbeat and vote path to remain responsive after the threading change, so that making the drive non-blocking does not degrade normal operation at supported cluster sizes.

Acceptance Criteria

1. While the cluster is healthy and all peers are connected, the controller shall send heartbeats at the configured interval without added latency attributable to the new dispatch path at supported cluster sizes (3–5 controllers).
2. The controller shall not introduce additional blocking or unbounded queuing on the steady-state heartbeat and vote send paths, which are already non-blocking enqueues.