myxce/.kiro/specs/ha-consistency-invariants/requirements.md

Requirements Document

Project Description (Input)

XCE HA adopts Raft as the consensus model and CP as the CAP stance (decision 2026-06-10, supersedes the earlier "No Raft" steering entry; single-writer log retained as XCE's simplification). Operators of multi-node XCE controller clusters currently get inconsistent durability guarantees: only WAL_JOB_SUBMIT has a majority-ACK commit barrier, while job state changes, queue control, worker control, and config push apply locally and replicate fire-and-forget; a newly elected primary force-promotes after a 60s timeout even with zero converged standbys; and quorum accounting has gate-only (TOCTOU) and catch-up-counting holes. This feature enforces the consistency invariants deduced from Raft + CP across the controller: (1) remove the CTRL_PRIMARY_COMMITTING state — election winners become CTRL_PRIMARY directly, with post-election write safety provided by the own-term commit rule (Raft §5.4.2) instead of a transitional state; (2) make majority ACK the hard commit barrier for ALL cluster-state-changing operations — reject with an error on timeout or quorum loss, never apply-and-degrade, with committed_lsn/committed_config_tag behaving as Raft commitIndex and state-machine apply only after commit; (3) keep full-consistency replication as best-effort — fan out to all standbys with leader-driven continuous catch-up streaming, majority being only the commit threshold. The universal commit barrier must be implemented before the COMMITTING state is removed. Full audit findings, file:line references, scope boundaries, and the Raft-rule mapping table are in brief.md; the Raft paper is the requirements authority and deviations must be documented with justification.

Introduction

The XCE controller cluster shall provide a uniform consistency contract governed by Raft consensus semantics with CP (consistency over availability) behavior. A cluster-state-changing operation is any operation that alters cluster state observable by clients or surviving a failover: job submission, job state transitions, queue control, worker membership changes, and configuration changes. The contract is: an operation is confirmed to its requester only after its log entry is durable on a majority of cluster members; an operation that cannot reach majority is reported as failed, never silently applied; and replication continuously targets all members, with majority as the commit threshold rather than the distribution target. Leadership is a single role — there is no transitional leader state with weaker guarantees. In-memory state is a provisional materialized view of the log: the persisted, majority-committed log (plus snapshots) is the authoritative cluster state, in-memory state must always be reconstructible from it, and anything externally consequential — confirmations to requesters and exports that drive external actions — is gated on commit, not on the in-memory view. The Raft paper (§5.3 commit rule, §5.4.2 own-term commit rule, Log Matching property) is the authority for these requirements; any deviation must be documented and justified in design.

Boundary Context

• In scope: Commit/confirmation semantics for all cluster-state-changing operations on the primary; leadership role unification (no transitional leader state); quorum accounting for accept and commit decisions; quorum-loss and standalone-mode behavior; leader-driven catch-up of lagging standbys.
• Out of scope: Vote and election correctness (owned by raft-library); replication wire format and standby-side apply mechanics (owned by ha-wal-replication); read consistency and leader leases (future work — reads may be stale during partition, and status queries may observe provisional in-memory state that is not yet committed; this is a documented deviation, while irrevocable-action exports remain commit-gated per criterion 2.4); automatic primary step-down on quorum loss (blocking writes satisfies CP; step-down is a possible follow-on); checkpoint replication (checkpoints are per-node by design, as Raft snapshots are); worker/scheduler reconnection after failover (worker-join-refactor).
• Adjacent expectations: This feature relies on the replication acknowledgment machinery introduced by ha-wal-replication and the heartbeat cadence defined by ha-heartbeat-liveness. Retained reported facts (criterion 2.8) cover facts the primary has received; facts lost before reaching the controller — or lost with the controller across a failover — are recovered by worker re-synchronization at rejoin (worker-rejoin-resync, future spec on the roadmap), which this feature does not own but is the designated backstop. The HA steering document (ha-async-io.md) describes the current two-tier model including the transitional leader state and must be synchronized after implementation.

Requirements

Requirement 1: Unified Primary Role

Objective: As a cluster operator, I want an election winner to act as primary immediately under a single leader role, so that clients are never redirected to a primary that does not exist and the cluster has no transitional mode with weaker consistency guarantees.

Acceptance Criteria

1. When a controller wins an election, the controller shall assume the primary role directly, without entering an intermediate transitional leader role.
2. The controller shall represent leadership as a single primary role; it shall not define a leader sub-state that accepts a different set of operations than the primary role.
3. While a controller is primary, when a client request requiring the primary arrives, the controller shall process the request subject to the commit barrier (Requirements 2 and 3) rather than redirecting the client to another node.
4. While a controller is primary, the controller shall report itself as the primary in status queries, regardless of whether it has yet committed an entry in its current term.
5. The controller shall not assume or change leadership based on a fixed wall-clock timeout that bypasses majority acknowledgment.

Requirement 2: Majority Commit Barrier for State-Changing Operations

Objective: As a cluster operator, I want every cluster-state-changing operation to be durable on a majority of members before it is confirmed, so that no single node failure or failover can lose an acknowledged change.

Acceptance Criteria

1. When the primary receives a cluster-state-changing operation (job submission, job state transition, queue control, or worker membership change), the primary shall confirm the operation to its requester only after the corresponding log entry has been acknowledged by a majority of cluster members, counting itself.
2. When a state-changing operation has a requester awaiting confirmation (such as a job submission or queue control request) and majority acknowledgment is not obtained within the configured commit timeout, the primary shall report the operation as failed to that requester.
3. If an operation has not achieved majority acknowledgment, the primary shall not advance its commit index past that operation's entry.
4. While a state-changing operation's log entry is not yet committed, the primary shall treat the resulting in-memory state as provisional: the primary shall not confirm the operation to its requester and shall not export the provisional state to consumers whose resulting actions cannot be revoked (such as scheduling and dispatch decisions).
5. If an operation was reported as failed due to commit timeout, the cluster shall converge on a single outcome for its log entry — committed on all members or discarded on all members — the primary shall never retroactively report that operation as successful to the original requester, and each member's in-memory state shall converge to that same outcome.
6. The controller shall advance its commit index monotonically; the commit index shall never decrease.
7. The controller shall be able to reconstruct its in-memory state from its persisted committed state (snapshot plus committed log entries), discarding any provisional state not present in the committed log.
8. When the primary receives a reported fact with no requester awaiting confirmation (such as a worker-reported job state transition), the primary shall retain the fact and continue attempting commitment — including across quorum-loss periods — until the fact's log entry is committed; while the primary remains running and holds leadership, it shall not discard an uncommitted reported fact.
9. (Facts-Apply invariant — added 2026-06-25 via /kiro-discovery.) When the primary applies a committed reported fact, the state-machine mutation and any externally-consequential reaction it triggers (worker slot release, scheduler dispatch, client/apiserver notification) shall occur only after the fact's log entry is committed, through the same post-commit apply seam used for requester-driven operations; the primary shall not perform a fact's externally-consequential reactions on receipt ahead of commit. This refines criterion 8 (which governs the durability/retention of facts) by governing the timing of their apply, and aligns facts with the Introduction principle that "anything externally consequential … is gated on commit, not on the in-memory view." Documented deviation (per the requirements authority's deviation rule): the current implementation applies WAL_MODE_RETAIN facts and fires their reactions on receipt (before commit). Reconciling the implementation with this criterion is owned by the raft-apply-decoupling spec — candidate plan and step list in .kiro/specs/raft-apply-decoupling/facts-in-raft.md — with the quorum-loss reaction policy tracked there as open decision D1 (block-but-durable vs a local-apply allowlist).

Requirement 3: Post-Election Write Safety

Objective: As a cluster operator, I want a newly elected primary to confirm operations only after an entry of its own term is committed, so that a leadership change never exposes uncommitted or divergent state from prior terms.

Acceptance Criteria

1. When a controller becomes primary, the primary shall not confirm any cluster-state-changing operation until at least one log entry created in its current term has been acknowledged by a majority of cluster members.
2. While the own-term commit condition of criterion 3.1 is unmet, if a state-changing request arrives, the primary shall either process it through the commit barrier (its entry satisfying criterion 3.1 upon majority acknowledgment) or reject it with an error indicating the cluster is not yet ready, and shall not confirm it by any other path.
3. While the own-term commit condition of criterion 3.1 is unmet, the primary shall not be subject to any timeout that confirms operations or advances the commit index without majority acknowledgment.

Requirement 4: Quorum Accounting Integrity

Objective: As a cluster operator, I want accept and commit decisions based on members that can actually receive and acknowledge entries, so that a write is never admitted or confirmed against a quorum that does not exist.

Acceptance Criteria

1. When the primary evaluates whether to accept a state-changing operation, it shall count toward quorum only cluster members currently able to receive new log entries; members that are disconnected or undergoing catch-up shall not be counted.
2. When the primary determines that an entry is committed, it shall require actual acknowledgments from a majority of cluster members; connectivity observed at acceptance time shall not substitute for acknowledgment.
3. If the set of reachable members shrinks below a majority after an operation is accepted but before majority acknowledgment is obtained, the primary shall report the operation as failed rather than confirm it.
4. While the cluster is configured with more than one member, the primary shall never treat zero peer acknowledgments as constituting a majority.

Requirement 5: Configuration Change Commit Barrier

Objective: As a cluster operator, I want configuration changes to follow the same majority-commit contract as other state changes, so that a failover never reverts a configuration upgrade that was reported as successful.

Acceptance Criteria

1. When the primary distributes a configuration change, the primary shall confirm the change to its requester only after a majority of cluster members have acknowledged it.
2. If majority acknowledgment for a configuration change is not obtained within the configured commit timeout, the primary shall report the configuration change as failed to its requester.
3. If a configuration change has not achieved majority acknowledgment, the primary shall not mark that configuration as committed.

Requirement 6: Full Replication and Leader-Driven Catch-Up

Objective: As a cluster operator, I want the primary to continuously bring all standbys current, so that the cluster trends toward full consistency and any standby can win a future election without losing committed state.

Acceptance Criteria

1. While holding leadership, the primary shall replicate new entries to all reachable standbys, not only to a majority subset.
2. When a standby falls behind the live replication stream, the primary shall stream the missing entries to that standby continuously, without requiring a heartbeat exchange to drive each replication step.
3. If a standby is behind the oldest retained log entry, the primary shall bring it current by transferring a snapshot followed by subsequent log entries.
4. When a lagging standby catches up to the primary's commit index, the primary shall re-include it in live replication.
5. While one or more standbys lag, the primary shall continue to confirm operations that achieve majority acknowledgment; full replication is best-effort and shall not become a confirmation precondition.

Requirement 7: Quorum Loss and Standalone Behavior

Objective: As a cluster operator, I want a primary that cannot reach majority to refuse state changes with a clear error and recover automatically, so that a partitioned cluster never diverges and no manual intervention is needed after the partition heals.

Acceptance Criteria

1. While the primary cannot reach a majority of cluster members, the primary shall reject cluster-state-changing operations with an error indicating that the cluster cannot currently commit.
2. When quorum loss occurs, the controller shall record the condition in its log so that operators can observe entry into and exit from the degraded state.
3. While quorum is lost, the primary shall continue to respond to status queries and replication-related traffic from returning members.
4. When majority connectivity is restored, the primary shall resume accepting state-changing operations without operator intervention.
5. While the cluster is configured with a single member, the controller shall treat its own acknowledgment as majority and confirm operations locally, preserving existing standalone behavior.