Rongsong Shen
59d9df246c
Per design invariant (a controller exits only on crash or admin kill), replace the self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual admin recovery). Records the never-self-terminate invariant as project-wide guidance in src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.
61 lines
10 KiB
Markdown
61 lines
10 KiB
Markdown
# Requirements Document
|
|
|
|
## Project Description (Input)
|
|
|
|
**Who has the problem:** Cluster operators running the XCE HA controller cluster — and every client whose acknowledged job/queue/worker operations must survive controller failover.
|
|
|
|
**Current situation:** In a 3-node failover test, committed jobs were permanently lost (`joblist` dropped 10→3; the 7 earliest, already acknowledged, vanished). Root cause (bug-661): **acknowledgment does not imply durability.** The controller advances its commit position on a majority of replication **acknowledgments**, but the primary counts its own contribution as soon as the entry is written to volatile buffers, not after it is persisted to stable storage — background persistence happens on a timer. A standby already persists before acknowledging, but the primary does not. So a "committed" operation can be durable on as few as one member; a single ill-timed crash + restart, or a restarted member with a truncated log winning leadership, discards it. This violates the documented assumption that a committed operation is held durably on a majority.
|
|
|
|
**What should change (chosen scope — durable commit + promotion fence):**
|
|
1. **Durable commit:** when the controller acknowledges an operation to a client, that operation's log entry must already be persisted to stable storage on a fault-tolerant majority (the acknowledging member plus ⌊N/2⌋ others). The controller must count its own copy toward the commit majority only once it is durably persisted.
|
|
2. **Recovery integrity and leadership fence (fail-loud):** a member that cannot reconstruct its committed state from persisted storage must be barred from leadership until it has caught up, and must never become authoritative while behind the cluster's known committed high-water mark — surfacing the failure loudly rather than silently discarding committed operations.
|
|
|
|
The mechanism (synchronous persistence vs. later group-commit batching) is a design decision; group-commit batching is deferred to the roadmap as a contract-preserving throughput optimization. See `.kiro/specs/ha-commit-durability/brief.md` for the full discovery analysis, evidence trail, and design decision.
|
|
|
|
## Introduction
|
|
|
|
This spec closes the durability gap beneath the HA consistency machinery: it makes acknowledgment imply durability. After this spec, any operation the controller reports as successful is guaranteed to be persisted on a majority of members at the moment of acknowledgment, and therefore survives the loss of any minority of members and any subsequent leadership change. It also fences a member that cannot recover its committed state — or that lags the cluster's committed high-water mark — from becoming the authoritative leader, so a truncated or corrupted log can never overwrite acknowledged operations. The client reply path already waits on the commit barrier; this spec strengthens the meaning of "committed" to "durably persisted on a majority," so the existing reply gate enforces the invariant. A testability provision makes the guarantee deterministically verifiable without depending on race timing.
|
|
|
|
## Boundary Context
|
|
|
|
- **In scope**: the durable-commit invariant on the primary's commit path (self contributes to the commit majority only when durably persisted); pinning the standby's persist-before-acknowledge behavior; the no-premature-acknowledgment guarantee; survival of acknowledged operations across the crash+restart of any minority; preserving cluster liveness (heartbeat/election) while performing durability work; recovery-integrity and leadership fencing for members that cannot reconstruct committed state or lag the committed high-water mark; degraded-quorum preservation of the invariant; a deterministic durability-verification provision; the soak E2E (`src/tools/test-commit-durability.sh`) plus unit coverage.
|
|
- **Out of scope**: the replication/acknowledgment wire format (`ha-wal-replication`); per-entry majority arithmetic and commit-index advancement (`ha-commit-engine` — consumed, this spec changes only the durability *precondition* on its inputs); demotion truncate/self-restart and rejoin reconciliation of an intact log (`ha-convergence-recovery`); promotion no-op and own-term gate (`ha-single-leader-role`); raw vote-grant arithmetic (`raft-library` — this spec adds a durability/high-water *precondition*, not new vote logic); WAL group-commit batching (roadmap — a contract-preserving throughput optimization, deferred); single-record job-array persistence (`job-array-single-wal-record`, roadmap); entity-bearing snapshots (roadmap); client-side controller failover (`libxce-failover`).
|
|
- **Adjacent expectations**: relies on `ha-commit-engine`'s commit-position semantics (monotonic, per-entry majority) and on `ha-wal-replication`'s acknowledgment and catch-up machinery; the client reply is already gated on the commit barrier (`ha-commit-barrier-rollout`), so strengthening "committed" to "durable-majority" enforces the invariant without changing the reply path. Downstream, `ha-convergence-recovery`'s single-outcome convergence depends on this spec making "persisted committed state" genuinely durable; a returning member that this spec fences from leadership until caught up complements that spec's truncate-and-restart reconciliation.
|
|
|
|
## Requirements
|
|
|
|
### Requirement 1: Durable Commit — Acknowledgment Implies Majority Persistence
|
|
|
|
**Objective:** As a cluster operator, on behalf of every client, I want any operation the controller acknowledges as successful to already be durably persisted on a majority of controllers, so that no acknowledged work is ever lost to a controller failure or failover.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When the controller acknowledges a state-changing operation to a client as successful, the controller shall have durably persisted that operation's log entry to stable storage on a majority of cluster members — the acknowledging member plus at least ⌊N/2⌋ other members.
|
|
2. While evaluating whether an operation is committed, the controller shall count its own log entry toward the commit majority only after that entry is durably persisted to its local stable storage, not when it is merely written to volatile buffers.
|
|
3. When a standby acknowledges a replicated log entry, the standby shall durably persist that entry to its local stable storage before sending the acknowledgment.
|
|
4. While a state-changing operation's log entry is not yet durably persisted on a majority of members, the controller shall not report that operation as successful to the requester.
|
|
5. If any minority of cluster members that contributed to an operation's commit are simultaneously lost and restarted, the acknowledged operation shall remain present and recoverable on the surviving cluster.
|
|
6. While performing durability persistence for a commit, the controller shall continue to service heartbeat and election processing without stalling, so that durability work does not induce spurious failover.
|
|
|
|
### Requirement 2: Recovery Integrity and Leadership Fence
|
|
|
|
**Objective:** As a cluster operator, I want a controller that cannot reconstruct its committed state to be barred from leadership until it has caught up, so that a member with a truncated or corrupted log can never become authoritative and discard acknowledged operations.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When the controller recovers its persisted log at startup, if the log cannot be reconstructed contiguously up to its last known committed boundary, the controller shall not silently discard committed entries and continue as if its committed state were complete.
|
|
2. If a controller's recovered log does not reach its last known committed boundary, the controller shall be ineligible to become primary until it has caught up to the cluster's committed boundary from a healthy member.
|
|
3. While a controller's committed position is below the cluster's known committed high-water mark, the controller shall not assume leadership in a way that would discard already-committed operations.
|
|
4. If a controller cannot recover to a consistent committed state and cannot catch up from a healthy member, the controller shall surface the condition as a loud, operator-visible alert and hold itself permanently non-promotable while staying alive and continuing to retry catch-up — refusing to serve potentially divergent state by refusing leadership, NOT by terminating the process. A controller MUST NEVER self-terminate (`exit`/`abort`) as a designed response to an operational condition: a self-exited controller requires manual admin recovery and degrades quorum meanwhile, whereas one that stays alive and keeps retrying self-heals automatically when a peer returns (see Requirement 2.6). (A genuine programming-invariant `assert`, or an external admin/supervisor kill, is not such a designed response.)
|
|
5. While the cluster operates with reduced redundancy or minimal quorum, the controller shall preserve the durable-commit guarantee of Requirement 1 across leadership changes, so that no acknowledged operation is lost even when quorum is temporarily degraded.
|
|
6. When a fenced member finishes catching up to the cluster's committed boundary, the controller shall restore that member's eligibility to participate in leadership.
|
|
|
|
### Requirement 3: Deterministic Durability Verification
|
|
|
|
**Objective:** As a developer verifying durability, I want the durability behavior to be deterministically controllable in a verification mode, so that the durable-before-acknowledge guarantee can be confirmed without depending on race timing.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. Where a durability-verification mode is enabled, the controller shall make the durability behavior deterministically controllable (for example, by configuring or disabling background persistence) so that an acknowledged operation's durability can be asserted without relying on timing.
|
|
2. Where durability-verification mode removes reliance on background persistence, an operation that the controller acknowledged as successful shall be recoverable after an immediate, ungraceful termination of a contributing member.
|
|
3. The controller shall preserve the same externally observable acknowledgment semantics whether or not durability-verification mode is enabled, so that the mode changes timing controllability only, not the durability contract.
|