myxce/.kiro/specs/ha-commit-engine/requirements.md

Requirements Document

Sub-spec 1 of 4 implementing the ha-consistency-invariants umbrella contract. The umbrella documents (.kiro/specs/ha-consistency-invariants/requirements.md, design.md, research.md) are the authoritative contract. This document restates the slice owned by this spec — umbrella criteria 2.1 (job-submit path), 2.3, 2.6, 4.1–4.4, 6.5, 7.1, 7.4, 7.5 — as this spec's own numbered requirements. Each criterion notes its umbrella ID; the traceability table at the end gives the full mapping.

Project Description (Input)

The majority-ACK check in the XCE controller is dishonest today: repl_wait_majority_ack counts standbys with acked_lsn > 0 ("ACKed anything ever"), not acked_lsn >= entry_lsn; quorum is checked only at the dispatch gate (TOCTOU); catch-up standbys count toward write quorum while excluded from broadcast; and zero active standbys short-circuits to success. Even the one existing commit barrier (job submit) provides no real per-entry guarantee. Cluster operators need a commit engine (commit.c / include/commit.h) that is the single authority for commit-index advancement: per-LSN majority evaluation (acked_lsn >= N, self counted), contiguous and monotonic committed_lsn, WAIT/RETAIN completion modes, an n_vcpus-relative admission cap, and membership-change re-evaluation — wired behind the existing job-submit sync path only, with the quorum-accounting holes in ha.c fixed.

Introduction

This spec delivers the commit engine and quorum-accounting integrity slice of the umbrella contract. After this spec, a job submission is confirmed to its requester only when its specific log entry has been acknowledged by a majority of cluster members (counting the primary itself); the commit index advances monotonically and contiguously, never on stale or substitute evidence; write admission counts only members actually able to receive entries; and a multi-member cluster never treats zero peer acknowledgments as a majority. Standalone (single-member) self-quorum behavior is preserved. The engine ships the full service interface (WAIT and RETAIN completion modes, own-term gate setter) so downstream sub-specs can consume it, but only the job-submit path is wired in this spec.

Boundary Context

• In scope: The commit engine (per-LSN majority evaluation, commit-index advancement, WAIT/RETAIN completion, admission cap, membership-change re-evaluation); the quorum-accounting fix in the dispatch gate (catch-up exclusion); feeding standby acknowledgments into the engine; removal of the zero-active-standbys success short-circuit for multi-member clusters; preservation of single-member self-quorum; unit/integration tests for all of the above.
• Out of scope: Classification and rollout of the barrier to other WAL types (ha-commit-barrier-rollout); own-term gate consumers and CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role — the commit_set_writable_lsn API ships here, but unset means no gate); demotion/egress-filter/catch-up changes (ha-convergence-recovery); config-change barrier error propagation (ha-commit-barrier-rollout); election and vote correctness (raft-library); replication wire format (ha-wal-replication).
• Adjacent expectations: Upstream, this spec relies on the ha-wal-replication machinery (per-standby acked_lsn tracking and WalAckReply processing — complete). Downstream, ha-commit-barrier-rollout, ha-single-leader-role, and ha-convergence-recovery consume the engine's service interface; changes to that interface require their revalidation.

Requirements

Requirement 1: Per-Entry Majority Commit for Job Submission

Objective: As a cluster operator, I want a job submission confirmed only after its specific log entry is durable on a majority of cluster members, so that no acknowledged submission can be lost by a single node failure or failover.

Acceptance Criteria

1. When the primary receives a job submission, the controller shall confirm the submission to its requester only after the submission's log entry has been acknowledged by a majority of cluster members, counting itself. (umbrella 2.1, job-submit path)
2. When the controller evaluates whether the entry with LSN N is committed, the controller shall count a standby's acknowledgment toward N only if that standby's acknowledged LSN is greater than or equal to N. (umbrella 4.2)
3. The controller shall not substitute connectivity observed at operation-acceptance time for actual per-entry acknowledgments when determining that an entry is committed. (umbrella 4.2)

Requirement 2: Commit Index Integrity

Objective: As a cluster operator, I want the commit index to advance only on real majority evidence, monotonically and without gaps, so that the committed prefix of the log is always a safe recovery point.

Acceptance Criteria

1. If an entry has not achieved majority acknowledgment, the controller shall not advance its commit index past that entry. (umbrella 2.3)
2. The controller shall advance its commit index monotonically; the commit index shall never decrease. (umbrella 2.6)
3. When acknowledgments arrive in any order, the controller shall advance the commit index contiguously — never beyond the lowest entry that lacks majority acknowledgment, and never beyond the highest locally appended entry. (umbrella 2.3, 2.6)

Requirement 3: Admission Quorum Accounting

Objective: As a cluster operator, I want write admission to count only cluster members that can actually receive new log entries, so that a write is never admitted against a quorum that does not exist.

Acceptance Criteria

1. When the primary evaluates whether to accept a state-changing operation, the controller shall count toward quorum only cluster members currently able to receive new log entries. (umbrella 4.1)
2. The controller shall exclude standbys undergoing catch-up from the write-admission quorum count. (umbrella 4.1)
3. The controller shall exclude disconnected members from the write-admission quorum count. (umbrella 4.1)

Requirement 4: Acknowledgment-Time Quorum Integrity

Objective: As a cluster operator, I want commit decisions re-verified at acknowledgment time against current membership, so that an operation accepted under a quorum that subsequently vanished is reported as failed rather than confirmed.

Acceptance Criteria

1. If the set of members able to acknowledge shrinks below a majority after an operation is accepted but before majority acknowledgment is obtained, the primary shall report the operation as failed rather than confirm it. (umbrella 4.3)
2. When cluster membership changes (standby connect, disconnect, catch-up entry, or catch-up completion), the controller shall re-evaluate pending commit waits against the updated membership. (umbrella 4.3)
3. While the cluster is configured with more than one member, the primary shall never treat zero peer acknowledgments as constituting a majority. (umbrella 4.4)
4. While the cluster is configured with more than one member and no standby acknowledgment covering the entry arrives within the commit timeout, the controller shall complete the commit wait with a failure result, not success. (umbrella 4.4 — removes the zero-active-standbys success short-circuit)

Requirement 5: Standby Lag Never Blocks Confirmation

Objective: As a cluster operator, I want majority to be the only commit threshold, so that one slow or lagging standby cannot stall confirmed writes.

Acceptance Criteria

1. While one or more standbys lag behind the replication stream, the primary shall continue to confirm operations that achieve majority acknowledgment; full replication shall not be a confirmation precondition. (umbrella 6.5)

Requirement 6: Quorum Loss, Restoration, and Standalone Behavior

Objective: As a cluster operator, I want a primary without majority to refuse state changes with a clear error and to resume automatically when majority returns, while a single-member cluster keeps confirming locally.

Acceptance Criteria

1. While the primary cannot reach a majority of cluster members, the primary shall reject cluster-state-changing operations with an error indicating that the cluster cannot currently commit. (umbrella 7.1)
2. When majority connectivity is restored, the primary shall resume accepting state-changing operations without operator intervention. (umbrella 7.4)
3. While the cluster is configured with a single member, the controller shall treat its own acknowledgment as majority and confirm operations locally, preserving existing standalone behavior. (umbrella 7.5)

Requirement 7: Commit-Wait Admission Control

Objective: As a cluster operator, I want the number of concurrently blocked commit waits bounded relative to my configured worker-thread budget, so that slow commits degrade into fast, retriable rejections instead of exhausting the request-processing capacity I tuned with --vcpus.

Acceptance Criteria

1. The controller shall bound the number of concurrent blocking commit waits relative to the configured worker-thread count (--vcpus), never as an absolute constant. (umbrella design §Commit Engine admission cap; locked decision, umbrella research item 3(d–e))
2. If admitting a new blocking commit wait would exceed the admission bound, the controller shall reject the operation with a retriable not-ready error instead of blocking. (umbrella design §Commit Engine admission cap)
3. While the admission bound is reached, the controller shall continue to process incoming acknowledgments and advance the commit index for already-admitted waits. (umbrella design §Commit Engine; ACK processing on the I/O thread is never blocked)

Umbrella Traceability

This spec	Umbrella ID	Notes
1.1	2.1	Job-submit path only; other operation types roll out in ha-commit-barrier-rollout
1.2, 1.3	4.2	Per-entry ACK verification (acked_lsn >= N)
2.1	2.3	No commit-index advance without majority
2.2	2.6	Monotonic commit index
2.3	2.3, 2.6	Contiguous advancement, bounded by appended log
3.1, 3.2, 3.3	4.1	Admission quorum counts only members able to receive; catch-up + disconnected excluded
4.1, 4.2	4.3	Post-acceptance quorum shrink → fail; membership-change re-evaluation
4.3, 4.4	4.4	Zero peer ACKs never a majority for n_peers > 1
5.1	6.5	Majority is the only commit threshold
6.1	7.1	Reject state changes without quorum
6.2	7.4	Auto-resume on quorum restoration
6.3	7.5	Standalone self-quorum preserved
7.1, 7.2, 7.3	umbrella design §Commit Engine (admission cap)	Locked design decision from umbrella research item 3(d–e); n_vcpus-relative, never absolute