Rongsong Shen
06135954ff
Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x), heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup. New: ha-consistency-invariants umbrella (requirements/design/research) and its four implementation sub-specs (specs generated via batch, cross-spec review applied), steering roadmap + product.md Raft+CP decision.
107 lines
11 KiB
Markdown
107 lines
11 KiB
Markdown
# Requirements Document
|
||
|
||
> Sub-spec 1 of 4 implementing the **ha-consistency-invariants** umbrella contract.
|
||
> The umbrella documents (`.kiro/specs/ha-consistency-invariants/requirements.md`, `design.md`, `research.md`) are the authoritative contract. This document restates the slice owned by this spec — umbrella criteria 2.1 (job-submit path), 2.3, 2.6, 4.1–4.4, 6.5, 7.1, 7.4, 7.5 — as this spec's own numbered requirements. Each criterion notes its umbrella ID; the traceability table at the end gives the full mapping.
|
||
|
||
## Project Description (Input)
|
||
|
||
The majority-ACK check in the XCE controller is dishonest today: `repl_wait_majority_ack` counts standbys with `acked_lsn > 0` ("ACKed anything ever"), not `acked_lsn >= entry_lsn`; quorum is checked only at the dispatch gate (TOCTOU); catch-up standbys count toward write quorum while excluded from broadcast; and zero active standbys short-circuits to success. Even the one existing commit barrier (job submit) provides no real per-entry guarantee. Cluster operators need a commit engine (`commit.c` / `include/commit.h`) that is the single authority for commit-index advancement: per-LSN majority evaluation (`acked_lsn >= N`, self counted), contiguous and monotonic `committed_lsn`, WAIT/RETAIN completion modes, an `n_vcpus`-relative admission cap, and membership-change re-evaluation — wired behind the existing job-submit sync path only, with the quorum-accounting holes in `ha.c` fixed.
|
||
|
||
## Introduction
|
||
|
||
This spec delivers the commit engine and quorum-accounting integrity slice of the umbrella contract. After this spec, a job submission is confirmed to its requester only when its specific log entry has been acknowledged by a majority of cluster members (counting the primary itself); the commit index advances monotonically and contiguously, never on stale or substitute evidence; write admission counts only members actually able to receive entries; and a multi-member cluster never treats zero peer acknowledgments as a majority. Standalone (single-member) self-quorum behavior is preserved. The engine ships the full service interface (WAIT and RETAIN completion modes, own-term gate setter) so downstream sub-specs can consume it, but only the job-submit path is wired in this spec.
|
||
|
||
## Boundary Context
|
||
|
||
- **In scope**: The commit engine (per-LSN majority evaluation, commit-index advancement, WAIT/RETAIN completion, admission cap, membership-change re-evaluation); the quorum-accounting fix in the dispatch gate (catch-up exclusion); feeding standby acknowledgments into the engine; removal of the zero-active-standbys success short-circuit for multi-member clusters; preservation of single-member self-quorum; unit/integration tests for all of the above.
|
||
- **Out of scope**: Classification and rollout of the barrier to other WAL types (ha-commit-barrier-rollout); own-term gate *consumers* and CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role — the `commit_set_writable_lsn` API ships here, but unset means no gate); demotion/egress-filter/catch-up changes (ha-convergence-recovery); config-change barrier error propagation (ha-commit-barrier-rollout); election and vote correctness (raft-library); replication wire format (ha-wal-replication).
|
||
- **Adjacent expectations**: Upstream, this spec relies on the ha-wal-replication machinery (per-standby `acked_lsn` tracking and WalAckReply processing — complete). Downstream, ha-commit-barrier-rollout, ha-single-leader-role, and ha-convergence-recovery consume the engine's service interface; changes to that interface require their revalidation.
|
||
|
||
## Requirements
|
||
|
||
### Requirement 1: Per-Entry Majority Commit for Job Submission
|
||
|
||
**Objective:** As a cluster operator, I want a job submission confirmed only after its specific log entry is durable on a majority of cluster members, so that no acknowledged submission can be lost by a single node failure or failover.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. When the primary receives a job submission, the controller shall confirm the submission to its requester only after the submission's log entry has been acknowledged by a majority of cluster members, counting itself. *(umbrella 2.1, job-submit path)*
|
||
2. When the controller evaluates whether the entry with LSN N is committed, the controller shall count a standby's acknowledgment toward N only if that standby's acknowledged LSN is greater than or equal to N. *(umbrella 4.2)*
|
||
3. The controller shall not substitute connectivity observed at operation-acceptance time for actual per-entry acknowledgments when determining that an entry is committed. *(umbrella 4.2)*
|
||
|
||
### Requirement 2: Commit Index Integrity
|
||
|
||
**Objective:** As a cluster operator, I want the commit index to advance only on real majority evidence, monotonically and without gaps, so that the committed prefix of the log is always a safe recovery point.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. If an entry has not achieved majority acknowledgment, the controller shall not advance its commit index past that entry. *(umbrella 2.3)*
|
||
2. The controller shall advance its commit index monotonically; the commit index shall never decrease. *(umbrella 2.6)*
|
||
3. When acknowledgments arrive in any order, the controller shall advance the commit index contiguously — never beyond the lowest entry that lacks majority acknowledgment, and never beyond the highest locally appended entry. *(umbrella 2.3, 2.6)*
|
||
|
||
### Requirement 3: Admission Quorum Accounting
|
||
|
||
**Objective:** As a cluster operator, I want write admission to count only cluster members that can actually receive new log entries, so that a write is never admitted against a quorum that does not exist.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. When the primary evaluates whether to accept a state-changing operation, the controller shall count toward quorum only cluster members currently able to receive new log entries. *(umbrella 4.1)*
|
||
2. The controller shall exclude standbys undergoing catch-up from the write-admission quorum count. *(umbrella 4.1)*
|
||
3. The controller shall exclude disconnected members from the write-admission quorum count. *(umbrella 4.1)*
|
||
|
||
### Requirement 4: Acknowledgment-Time Quorum Integrity
|
||
|
||
**Objective:** As a cluster operator, I want commit decisions re-verified at acknowledgment time against current membership, so that an operation accepted under a quorum that subsequently vanished is reported as failed rather than confirmed.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. If the set of members able to acknowledge shrinks below a majority after an operation is accepted but before majority acknowledgment is obtained, the primary shall report the operation as failed rather than confirm it. *(umbrella 4.3)*
|
||
2. When cluster membership changes (standby connect, disconnect, catch-up entry, or catch-up completion), the controller shall re-evaluate pending commit waits against the updated membership. *(umbrella 4.3)*
|
||
3. While the cluster is configured with more than one member, the primary shall never treat zero peer acknowledgments as constituting a majority. *(umbrella 4.4)*
|
||
4. While the cluster is configured with more than one member and no standby acknowledgment covering the entry arrives within the commit timeout, the controller shall complete the commit wait with a failure result, not success. *(umbrella 4.4 — removes the zero-active-standbys success short-circuit)*
|
||
|
||
### Requirement 5: Standby Lag Never Blocks Confirmation
|
||
|
||
**Objective:** As a cluster operator, I want majority to be the only commit threshold, so that one slow or lagging standby cannot stall confirmed writes.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. While one or more standbys lag behind the replication stream, the primary shall continue to confirm operations that achieve majority acknowledgment; full replication shall not be a confirmation precondition. *(umbrella 6.5)*
|
||
|
||
### Requirement 6: Quorum Loss, Restoration, and Standalone Behavior
|
||
|
||
**Objective:** As a cluster operator, I want a primary without majority to refuse state changes with a clear error and to resume automatically when majority returns, while a single-member cluster keeps confirming locally.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. While the primary cannot reach a majority of cluster members, the primary shall reject cluster-state-changing operations with an error indicating that the cluster cannot currently commit. *(umbrella 7.1)*
|
||
2. When majority connectivity is restored, the primary shall resume accepting state-changing operations without operator intervention. *(umbrella 7.4)*
|
||
3. While the cluster is configured with a single member, the controller shall treat its own acknowledgment as majority and confirm operations locally, preserving existing standalone behavior. *(umbrella 7.5)*
|
||
|
||
### Requirement 7: Commit-Wait Admission Control
|
||
|
||
**Objective:** As a cluster operator, I want the number of concurrently blocked commit waits bounded relative to my configured worker-thread budget, so that slow commits degrade into fast, retriable rejections instead of exhausting the request-processing capacity I tuned with `--vcpus`.
|
||
|
||
#### Acceptance Criteria
|
||
|
||
1. The controller shall bound the number of concurrent blocking commit waits relative to the configured worker-thread count (`--vcpus`), never as an absolute constant. *(umbrella design §Commit Engine admission cap; locked decision, umbrella research item 3(d–e))*
|
||
2. If admitting a new blocking commit wait would exceed the admission bound, the controller shall reject the operation with a retriable not-ready error instead of blocking. *(umbrella design §Commit Engine admission cap)*
|
||
3. While the admission bound is reached, the controller shall continue to process incoming acknowledgments and advance the commit index for already-admitted waits. *(umbrella design §Commit Engine; ACK processing on the I/O thread is never blocked)*
|
||
|
||
## Umbrella Traceability
|
||
|
||
| This spec | Umbrella ID | Notes |
|
||
|-----------|-------------|-------|
|
||
| 1.1 | 2.1 | Job-submit path only; other operation types roll out in ha-commit-barrier-rollout |
|
||
| 1.2, 1.3 | 4.2 | Per-entry ACK verification (`acked_lsn >= N`) |
|
||
| 2.1 | 2.3 | No commit-index advance without majority |
|
||
| 2.2 | 2.6 | Monotonic commit index |
|
||
| 2.3 | 2.3, 2.6 | Contiguous advancement, bounded by appended log |
|
||
| 3.1, 3.2, 3.3 | 4.1 | Admission quorum counts only members able to receive; catch-up + disconnected excluded |
|
||
| 4.1, 4.2 | 4.3 | Post-acceptance quorum shrink → fail; membership-change re-evaluation |
|
||
| 4.3, 4.4 | 4.4 | Zero peer ACKs never a majority for n_peers > 1 |
|
||
| 5.1 | 6.5 | Majority is the only commit threshold |
|
||
| 6.1 | 7.1 | Reject state changes without quorum |
|
||
| 6.2 | 7.4 | Auto-resume on quorum restoration |
|
||
| 6.3 | 7.5 | Standalone self-quorum preserved |
|
||
| 7.1, 7.2, 7.3 | umbrella design §Commit Engine (admission cap) | Locked design decision from umbrella research item 3(d–e); `n_vcpus`-relative, never absolute |
|