Rongsong Shen
06135954ff
Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x), heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup. New: ha-consistency-invariants umbrella (requirements/design/research) and its four implementation sub-specs (specs generated via batch, cross-spec review applied), steering roadmap + product.md Raft+CP decision.
92 lines
12 KiB
Markdown
92 lines
12 KiB
Markdown
# Requirements Document
|
|
|
|
> Sub-spec 2 of 4 implementing the **ha-consistency-invariants** umbrella contract.
|
|
> The umbrella documents (`.kiro/specs/ha-consistency-invariants/requirements.md`, `design.md`, `research.md`) are the authoritative contract. This document restates the slice owned by this spec — umbrella criteria 2.1 (all operation types), 2.2, 2.8, 5.1, 5.2, 5.3 — as this spec's own numbered requirements. Each criterion notes its umbrella ID; the traceability table at the end gives the full mapping. Locked umbrella decisions (log-is-the-queue for facts, commit-first-then-reply for all request-shaped operations, crash-loss of uncommitted facts accepted until worker-rejoin-resync) are restated, never re-derived.
|
|
|
|
## Project Description (Input)
|
|
|
|
Only WAL_JOB_SUBMIT goes through a commit barrier today. Job state changes, queue control, and worker control apply locally with fire-and-forget replication; job/queue/worker control handlers reply to their requesters before any durability is established; config push ignores its majority-ACK result and confirms the upgrade to the admin tool regardless; worker-reported facts are silently lost when the dispatch gate rejects them (bug-396 family); and `mark_job_*` transitions have no idempotency guards, so duplicate or replayed application corrupts state. Cluster operators need every state-changing operation routed per the umbrella design's write-path classification table: request-shaped operations confirm to their requester only after majority commit (validate → mutate provisional → append → commit barrier → reply) and fail to the requester on commit timeout; fact-shaped operations are appended and retained until committed with no timeout — never dropped while the primary runs and leads; configuration upgrades fail to the requester on missed majority; and fact transitions are idempotent and ordered so retry/replay is safe. This spec consumes the commit engine delivered by ha-commit-engine (commit_submit WAIT/RETAIN) and never modifies the engine's internals.
|
|
|
|
## Introduction
|
|
|
|
This spec delivers the barrier-rollout slice of the umbrella contract. After this spec, every cluster-state-changing operation on the primary participates in the majority-commit contract: a requester (client, scheduler, worker, or admin tool) that receives a success reply is guaranteed the operation is durable on a majority of cluster members; a requester whose operation cannot reach majority within the commit timeout receives an error; and reported facts with no requester awaiting confirmation are retained in the log and retried until committed — including across quorum-loss periods. The write path enforces this through an exhaustive classification of WAL entry types: a state-changing entry type that is not classified cannot silently bypass the barrier. Configuration changes follow the same contract on their own track. Job state transitions become idempotent and ordered so that retained-fact retry, replay, and the future worker-rejoin-resync can re-apply them safely.
|
|
|
|
## Boundary Context
|
|
|
|
- **In scope**: The write-path classification table and its dispatch at the persistence seam (with a fail-fast guard for unclassified types); commit-first-then-reply restructuring of every request-shaped control handler (job control, queue control, worker control, worker join/update) and verification of every control-path reply site; classification verdict for the scheduler dispatch acknowledgment; RETAIN routing for fact-shaped WAL types, including acceptance of reported facts during quorum loss; configuration-change commit-barrier error propagation to the upgrade requester; idempotency and ordering guards for job state transitions; integration tests proving no success reply is observable before commit and that facts survive quorum-loss periods.
|
|
- **Out of scope**: The commit engine itself — per-LSN evaluation, admission cap, `committed_lsn` advancement (ha-commit-engine, consumed as-is); promotion no-op, own-term gate consumers, and CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role — must land after this spec); demotion truncate+restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum-transition logging (ha-convergence-recovery); worker protocol changes of any kind — facts that never reached the controller, or that died with it across failover, are recovered only by worker-rejoin-resync (roadmap backlog, designated backstop); election and vote correctness (raft-library); replication wire format and standby apply mechanics (ha-wal-replication).
|
|
- **Adjacent expectations**: Upstream, this spec consumes the `commit_submit` WAIT/RETAIN service interface and the quorum-accounting fixes delivered by ha-commit-engine; changes to that interface require revalidation here. Downstream, ha-single-leader-role depends on this spec having placed the barrier on all write paths before the transitional leader state is removed (hard ordering), and worker-rejoin-resync shares the idempotency guards introduced here. Crash-loss of facts that were appended but not yet committed when the primary process dies is accepted until worker-rejoin-resync lands (umbrella-locked decision).
|
|
|
|
## Requirements
|
|
|
|
### Requirement 1: Commit-First-Then-Reply for Request-Shaped Operations
|
|
|
|
**Objective:** As a cluster operator, I want every operation with a requester awaiting confirmation to be confirmed only after its log entries are durable on a majority of cluster members, so that no acknowledged change can be lost by a single node failure or failover.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When the primary receives a request-shaped cluster-state-changing operation (job control, queue control, worker control, worker join, or worker update), the primary shall send a success reply to the requester only after every log entry produced by the operation has been acknowledged by a majority of cluster members, counting itself. *(umbrella 2.1)*
|
|
2. If majority acknowledgment for a request-shaped operation is not obtained within the configured commit timeout, the primary shall reply to that requester with an error reporting the operation as failed. *(umbrella 2.2)*
|
|
3. If the commit barrier rejects a request-shaped operation without waiting (cluster not ready to commit), the primary shall reply to the requester with the corresponding error, never with success. *(umbrella 2.2)*
|
|
4. While the log entries produced by a request-shaped operation are not yet committed, the primary shall not emit any reply that the requester can interpret as confirmation of that operation. *(umbrella 2.1)*
|
|
|
|
### Requirement 2: Complete Write-Path Classification
|
|
|
|
**Objective:** As a cluster operator, I want every state-changing entry type explicitly classified as request-shaped or fact-shaped at the write path, so that no operation type can silently bypass the commit contract.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. The controller shall route every state-changing log entry type through the commit engine under an explicit classification: request-shaped types complete through the commit barrier before requester confirmation, and fact-shaped types are retained until committed. *(umbrella 2.1, 2.8)*
|
|
2. If a state-changing log entry type with no classification reaches the write path, the controller shall fail fast at that point in testing builds; in production builds it shall log an error and treat the entry as fact-shaped, never dropping it and never confirming it early. *(umbrella 2.1)*
|
|
3. When the scheduler delivers a job dispatch message, the primary's immediate acknowledgment shall constitute only receipt of the message; per-job dispatch outcomes shall continue to be delivered through the existing per-job notifications, and the receipt acknowledgment shall not confirm any state-changing log entry. *(umbrella 2.1)*
|
|
4. The controller shall contain no control-path reply site that confirms a request-shaped state-changing operation before its entries are committed; every control-path reply site shall be either a post-commit confirmation or a receipt-only acknowledgment for fact-shaped deliveries. *(umbrella 2.1)*
|
|
|
|
### Requirement 3: Fact Retention — Log Is the Queue
|
|
|
|
**Objective:** As a cluster operator, I want reported facts that have no requester awaiting confirmation to be retained and retried until committed, so that a quorum-loss window never silently erases what actually happened on the cluster.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When the primary receives a reported fact with no requester awaiting confirmation (a worker-reported job state transition, worker-reported resource usage, scheduler-reported pending reason, or controller-detected worker death), the primary shall append the fact to its log and continue attempting commitment until the fact's entry is committed, with no commit timeout. *(umbrella 2.8)*
|
|
2. While the primary cannot reach a majority of cluster members, when a reported fact arrives, the primary shall accept and append the fact rather than rejecting it. *(umbrella 2.8)*
|
|
3. While the primary remains running and holds leadership, the primary shall not discard an uncommitted reported fact. *(umbrella 2.8)*
|
|
4. When majority connectivity is restored, the primary shall bring previously appended uncommitted facts to committed state without requiring redelivery from the original reporter. *(umbrella 2.8)*
|
|
|
|
### Requirement 4: Configuration Change Commit Barrier
|
|
|
|
**Objective:** As a cluster operator, I want configuration changes to follow the same majority-commit contract as other state changes, so that a failover never reverts a configuration upgrade that was reported as successful.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When the primary distributes a configuration change, the primary shall confirm the change to its requester only after a majority of cluster members have acknowledged it. *(umbrella 5.1)*
|
|
2. If majority acknowledgment for a configuration change is not obtained within the configured commit timeout, the primary shall report the configuration change as failed to its requester. *(umbrella 5.2)*
|
|
3. If a configuration change has not achieved majority acknowledgment, the primary shall not mark that configuration as committed. *(umbrella 5.3)*
|
|
4. If a configuration change was reported as failed to its requester, the primary shall not proceed with the restart that activates the new configuration. *(umbrella 5.2)*
|
|
|
|
### Requirement 5: Idempotent and Ordered Fact Application
|
|
|
|
**Objective:** As a cluster operator, I want job state transitions to be safe under duplicate and replayed application, so that retained-fact retry, log replay, and future worker rejoin re-synchronization cannot corrupt job state.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When a job state transition that is already reflected in the job's current state is applied again, the controller shall leave the job's state unchanged and record the duplicate application in its log. *(umbrella 2.8)*
|
|
2. If a job state transition would move a job from a terminal state back to a non-terminal state, the controller shall reject the transition, preserve the terminal state, and log a warning. *(umbrella 2.8)*
|
|
3. When a fact's log entry is applied more than once through retry or replay, the resulting job state shall equal the state produced by exactly one application. *(umbrella 2.8)*
|
|
|
|
## Umbrella Traceability
|
|
|
|
| This spec | Umbrella ID | Notes |
|
|
|-----------|-------------|-------|
|
|
| 1.1, 1.4 | 2.1 | Commit-first-then-reply generalized from job submit to all request-shaped operation types |
|
|
| 1.2, 1.3 | 2.2 | Requester-awaiting operations fail on commit timeout / not-ready, never silently degrade |
|
|
| 2.1 | 2.1, 2.8 | Classification table is the enforcement mechanism for "all operation types" |
|
|
| 2.2 | 2.1 | Fail-fast guard: a new/unclassified WAL type cannot silently bypass the barrier (umbrella revalidation trigger) |
|
|
| 2.3 | 2.1 | handle_jobdispatch classification per the umbrella Reply-Ordering Contract (receipt-ACK verdict, task-verified) |
|
|
| 2.4 | 2.1 | Exhaustive reply-site sweep; a missed handler silently violates umbrella 2.1 |
|
|
| 3.1, 3.3 | 2.8 | Log-is-the-queue: RETAIN mode, no timeout, retry until committed (umbrella-locked decision) |
|
|
| 3.2 | 2.8 | Facts accepted across quorum-loss periods (closes the bug-396 silent-loss hole) |
|
|
| 3.4 | 2.8 | Commitment resumes from the retained log suffix, not from reporter redelivery |
|
|
| 4.1 | 5.1 | Config change confirmed only after majority ACK |
|
|
| 4.2, 4.4 | 5.2 | Config change fails to requester on timeout; restart sequencing gated on the barrier (roadmap residual) |
|
|
| 4.3 | 5.3 | committed_config_tag advances only on majority ACK |
|
|
| 5.1, 5.2, 5.3 | 2.8 | Idempotency guards make RETAIN retry/replay safe; shared prerequisite of worker-rejoin-resync |
|