myxce/.kiro/specs/ha-single-leader-role/requirements.md

Requirements Document

Sub-spec 3 of 4 implementing the ha-consistency-invariants umbrella contract. The umbrella documents (.kiro/specs/ha-consistency-invariants/requirements.md, design.md, research.md) are the authoritative contract. This document restates the slice owned by this spec — umbrella criteria 1.1, 1.2, 1.3, 1.4, 1.5, 3.1, 3.2, 3.3 — as this spec's own numbered requirements. Each criterion notes its umbrella ID; the traceability table at the end gives the full mapping. Locked umbrella decisions (own-term commit via an explicit no-op log entry appended at promotion; no wall-clock timeout may change leadership or confirm operations; the universal commit barrier must be in force before the transitional state is removed) are restated, never re-derived. HARD ORDERING: this spec must land AFTER ha-commit-barrier-rollout. Removing the transitional state before the universal barrier is in force opens a window where a fresh primary accepts unreplicated writes.

Project Description (Input)

CTRL_PRIMARY_COMMITTING is a transitional leader state that Raft does not have. It causes a redirect loop (ha_should_redirect points clients at a primary that does not exist yet), committing_timeout_fn force-promotes after 60 seconds even with zero converged standbys (violating the commit rule outright), and ha_is_primary reports "not primary" during the window even though this node won the election and no other primary exists. Cluster operators need Raft's three-state model: an election winner becomes CTRL_PRIMARY directly; post-election write safety comes from committing a no-op log entry of the current term (Raft §5.4.2 own-term commit rule) through the commit engine's writable-LSN gate; and no wall-clock path ever changes leadership or confirms operations. The transitional state, its force-promotion watchdog, its convergence counter, and its acknowledgment counters are deleted outright, with every reference across the controller, its tests, and the steering documentation swept to zero.

Introduction

This spec delivers the single-leader-role slice of the umbrella contract. After this spec, leadership in the XCE controller cluster is a single role: a controller that wins an election is the primary immediately — it reports itself as primary, accepts client traffic subject to the commit barrier, and is never the target of a redirect loop. The weaker guarantee that the transitional state was trying to provide ("don't confirm anything until the cluster has converged") is provided correctly instead by the own-term commit rule: at promotion the new primary appends a no-op entry of its current term and withholds confirmation of every operation until that entry is committed on a majority. Because the entry commits only through real majority acknowledgment, there is no timeout that can force the gate open — the 60-second force-promotion watchdog and the convergence/acknowledgment counting that fed it are removed without replacement. The dispatch-gate behavior contributed by upstream specs (including acceptance of reported facts during quorum loss) is preserved unchanged.

Boundary Context

• In scope: Direct election-winner promotion to the primary role; the promotion-time no-op log entry of the current term and its use as the own-term confirmation gate; removal of the transitional leader state, its force-promotion timeout, its convergence checker, and its acknowledgment counters; honest primary self-reporting and redirect behavior; symmetric update of the legacy (testing-only) election evaluation path; exhaustive sweep of every transitional-state reference across controller code, test fixtures, and steering/contributor documentation, verified to zero.
• Out of scope: Election and vote correctness — vote RPCs, term management, freshness gates (raft-library); commit-engine internals — per-entry majority evaluation, the writable-LSN gate mechanics, admission cap (ha-commit-engine, consumed as-is); operation classification and reply ordering — which operations wait and which are retained (ha-commit-barrier-rollout, must already be in force); demotion behavior — truncate-and-restart, rejoin reconciliation, egress filtering, catch-up driving, quorum-transition logging (ha-convergence-recovery); replication wire format and standby apply mechanics (ha-wal-replication).
• Adjacent expectations: Upstream, this spec consumes the commit engine's writable-LSN gate (commit_set_writable_lsn; the engine withholds waiter completion until the commit index covers the writable LSN) delivered by ha-commit-engine, and requires the universal commit barrier plus the dispatch-gate fact allowlist delivered by ha-commit-barrier-rollout to be in force — the gate simplification here must preserve that allowlist. Downstream, ha-convergence-recovery's end-to-end failover validation relies on the simplified lifecycle, and the HA steering document (ha-async-io.md) — whose dispatch-gate table and replication-consistency section still describe the transitional state — is synchronized by this spec.

Requirements

Requirement 1: Unified Primary Role

Objective: As a cluster operator, I want an election winner to act as primary immediately under a single leader role, so that clients are never redirected to a primary that does not exist and the cluster has no transitional mode with weaker consistency guarantees.

Acceptance Criteria

1. When a controller wins an election, the controller shall assume the primary role directly, without entering an intermediate transitional leader role. (umbrella 1.1)
2. The controller shall represent leadership as a single primary role; it shall not define a leader sub-state that accepts a different set of operations than the primary role, and no transitional leader state shall remain observable anywhere in the controller, its tests, or its operator-facing documentation. (umbrella 1.2)
3. While a controller is primary, when a client request requiring the primary arrives, the controller shall process the request subject to the commit barrier rather than redirecting the client to another node. (umbrella 1.3)
4. While a controller is primary, the controller shall report itself as the primary in status queries, regardless of whether it has yet committed an entry in its current term. (umbrella 1.4)
5. The controller shall not assume or change leadership based on a fixed wall-clock timeout that bypasses majority acknowledgment. (umbrella 1.5)

Requirement 2: Post-Election Write Safety

Objective: As a cluster operator, I want a newly elected primary to confirm operations only after an entry of its own term is committed, so that a leadership change never exposes uncommitted or divergent state from prior terms.

Acceptance Criteria

1. When a controller becomes primary, the primary shall not confirm any cluster-state-changing operation until at least one log entry created in its current term has been acknowledged by a majority of cluster members; the primary shall create such an entry at promotion so that an idle cluster's new primary becomes writable without waiting for client traffic. (umbrella 3.1)
2. While the own-term commit condition of criterion 2.1 is unmet, if a state-changing request arrives, the primary shall either process it through the commit barrier (its entry satisfying criterion 2.1 upon majority acknowledgment) or reject it with an error indicating the cluster is not yet ready, and shall not confirm it by any other path. (umbrella 3.2)
3. While the own-term commit condition of criterion 2.1 is unmet, the primary shall not be subject to any timeout that confirms operations or advances the commit index without majority acknowledgment. (umbrella 3.3)

Umbrella Traceability

This spec	Umbrella ID	Notes
1.1	1.1	Election winner → primary directly; transitional state never entered
1.2	1.2	Single leader role; transitional state deleted and swept to zero references (code, tests, docs)
1.3	1.3	No redirect from a live primary; the redirect-loop window cannot exist once the state is gone
1.4	1.4	Status honesty: primary self-reports as primary before its own-term entry commits
1.5	1.5	Force-promotion watchdog deleted without replacement; no wall-clock leadership change
2.1	3.1	Own-term commit rule (Raft §5.4.2) via explicit promotion no-op entry (umbrella-locked decision)
2.2	3.2	Pre-own-term requests: through the barrier or not-ready rejection — never a third path
2.3	3.3	No timeout substitutes for majority acknowledgment while the own-term gate is unmet