179 lines
12 KiB
Markdown
179 lines
12 KiB
Markdown
# Requirements Document
|
|
|
|
## Introduction
|
|
|
|
This feature makes XCE controller primary failover transparent to client applications built on
|
|
libxce (the XCE client library), including the worker, scheduler, and API server daemons and
|
|
the `xce_test` REPL.
|
|
|
|
Today, when the connected controller primary is killed in an HA cluster, libxce keeps using the
|
|
dead connection and returns an opaque invalid-argument error indefinitely: it never fails over
|
|
to the newly elected primary and never reports a clear "cluster not ready" during the election
|
|
window. The upstream requirement (`ha-config-bootstrap` Requirement 7) delivered only a
|
|
standalone retry primitive that no production request path invokes and that cannot discover a
|
|
new primary after the connected one dies. This spec completes that intent on the client side:
|
|
it routes all client requests through transparent retry/failover, rotates the configured master
|
|
list to locate the new primary, and surfaces a distinct cluster-not-ready error only when an
|
|
election outlasts the retry budget.
|
|
|
|
## Boundary Context
|
|
|
|
- **In scope**: client-side transparent failover for all libxce request types; discovery of the
|
|
new primary by rotating the configured master list, using the redirect address (when present) as
|
|
a best-effort hint to try first; bounded transparent retry that surfaces a distinct
|
|
cluster-not-ready error on budget exhaustion; failover for both read and write requests;
|
|
configurable retry/backoff parameters with safe defaults; operator-visible logging of failover
|
|
events; one small controller change so that redirect responses carry the primary's host and port
|
|
(the wire schema and reply struct already define these fields); end-to-end verification with the
|
|
`xce_test` client (the controller test harness, and the original reproduction case for this bug).
|
|
- **Out of scope**: controller-side election, log replication, commit barrier, and write
|
|
idempotency (consumed, not changed); introducing new probe or transport message types or new
|
|
reply fields (the redirect address fields already exist); per-request retry opt-in flags (writes
|
|
fail over by default); **adoption by the worker, scheduler, and apiserver daemons** — their
|
|
master-list configuration and liveness-safe handling of bounded blocking are owned by the
|
|
follow-on spec `component-failover-adoption` (depends on this spec).
|
|
- **Adjacent expectations**: libxce classifies controller responses by the distinct statuses the
|
|
controller actually emits — a *redirect* status (a primary exists but this node is a standby), a
|
|
*no-primary* status (this standby knows no primary yet), a *not-ready* status (the controller is
|
|
still initializing or electing), and a *standby* status (the operation is primary-only). With
|
|
the in-scope controller change, a redirect response carries the primary's host and port; libxce
|
|
treats this as a **best-effort hint** — it tries the hinted address first, and rotates its
|
|
configured master list when the hint is absent, stale, or unreachable. Correctness never depends
|
|
on the hint. libxce relies on controllers to deduplicate re-sent
|
|
writes through their commit-barrier idempotency guards, and on the configured master list (e.g.
|
|
`set masters host:port,host:port,host:port`) being the full set of controller candidates.
|
|
- **Stateful consumers**: a pure client (`xce_test`) needs only reconnect + resend. Stateful daemons
|
|
(apiserver, worker, scheduler) hold a session established via join; after a failover they must
|
|
re-establish it on the new primary (and a worker re-syncs its running jobs). libxce provides a
|
|
reconnect/rejoin hook for this; the hook *content* is owned by `component-failover-adoption` and
|
|
`worker-rejoin-resync`. Fire-and-forget notifications have no reply, so libxce cannot detect a
|
|
standby for them or guarantee delivery across a failover — redelivery is `worker-rejoin-resync`.
|
|
|
|
## Requirements
|
|
|
|
### Requirement 1: Transparent Failover for All Client Requests
|
|
|
|
**Objective:** As a developer of a libxce-based daemon, I want every controller request to fail
|
|
over automatically inside the library, so that I do not write ad-hoc retry or reconnect loops in
|
|
caller code.
|
|
|
|
#### Acceptance Criteria
|
|
1. The libxce client library shall route every controller request, both read/query and
|
|
mutating/write requests, through the transparent retry-and-failover path.
|
|
2. When controller failover occurs and a new primary becomes available, the libxce client
|
|
library shall complete the next request against the new primary without requiring any change
|
|
to caller code.
|
|
3. While failover handling is active, the libxce client library shall preserve the existing
|
|
public API call signatures and success/error return conventions so that existing callers
|
|
continue to compile and run unchanged.
|
|
|
|
### Requirement 2: New-Primary Discovery Across the Configured Master List
|
|
|
|
**Objective:** As a client application, I want libxce to find the new primary after the connected
|
|
one dies, so that my requests continue to reach an authoritative controller.
|
|
|
|
#### Acceptance Criteria
|
|
1. If the connected controller becomes unreachable through connection refused, connection
|
|
timeout, or peer reset, the libxce client library shall attempt the other controllers in its
|
|
configured master list to locate the current primary.
|
|
2. When a contacted controller responds with a redirect status indicating that a primary exists
|
|
elsewhere, the libxce client library shall locate the primary by rotating its configured master
|
|
list, trying the redirect address (when the response carries one) first as a best-effort hint.
|
|
3. When a contacted controller responds that the requested operation is primary-only (a standby
|
|
status), the libxce client library shall treat it as a redirect and attempt the other
|
|
controllers in its configured master list.
|
|
4. When a controller accepts and serves the request as primary, the libxce client library shall
|
|
treat it as the current primary and direct subsequent requests to it.
|
|
5. The libxce client library shall not consider a request successful when it was answered by a
|
|
controller reporting a redirect, standby, no-primary, or not-ready status.
|
|
|
|
### Requirement 3: Election-Window Behavior and Cluster-Not-Ready Reporting
|
|
|
|
**Objective:** As a client application, I want short failovers hidden and longer outages reported
|
|
clearly, so that I can distinguish "retry later" from a real failure.
|
|
|
|
#### Acceptance Criteria
|
|
1. While every reachable controller in the configured master list reports no primary exists (a
|
|
no-primary or not-ready status), the libxce client library shall retry across the configured
|
|
controllers with exponential backoff until the retry budget is exhausted.
|
|
2. When a primary becomes available before the retry budget is exhausted, the libxce client
|
|
library shall complete the request successfully and return no error to the caller.
|
|
3. If the retry budget is exhausted while no controller has become primary, the libxce client
|
|
library shall return a distinct cluster-not-ready error to the caller.
|
|
4. The cluster-not-ready error shall be distinguishable by the caller from transport connection
|
|
errors and from request-validation errors, so that a no-primary window is never reported as an
|
|
opaque invalid-argument error.
|
|
|
|
### Requirement 4: Error Classification — Retriable vs. Terminal
|
|
|
|
**Objective:** As a client application, I want transient controller errors retried and permanent
|
|
errors surfaced immediately, so that I neither hang on unrecoverable errors nor fail on recoverable ones.
|
|
|
|
#### Acceptance Criteria
|
|
1. If a controller returns a no-primary or not-ready status, the libxce client library shall retry
|
|
the request with exponential backoff.
|
|
2. If a controller returns a redirect status (a primary exists elsewhere), the libxce client
|
|
library shall locate and switch to the primary per Requirement 2 and retry promptly, without
|
|
waiting for the full backoff delay used for the no-primary case.
|
|
3. If a controller returns a standby status (the operation is primary-only), the libxce client
|
|
library shall switch to another configured controller per Requirement 2 and retry.
|
|
4. If a transport-level connection refused, timeout, or peer reset occurs, the libxce client
|
|
library shall reconnect or rotate to another configured controller and retry.
|
|
5. If a controller returns a non-retriable error, specifically authentication failure, permission
|
|
denied, invalid request, or not found, the libxce client library shall surface that error to
|
|
the caller immediately without retry.
|
|
|
|
### Requirement 5: Read and Write Failover Semantics
|
|
|
|
**Objective:** As a client application, I want both queries and mutating operations to survive
|
|
failover without duplicating side effects, so that failover is safe as well as transparent.
|
|
|
|
#### Acceptance Criteria
|
|
1. The libxce client library shall apply transparent failover to both read/query requests and
|
|
mutating/write requests, such as job submission.
|
|
2. When a write request is retried after a failover, the libxce client library shall not cause the
|
|
operation to be applied more than once, relying on the controller's idempotency handling for
|
|
re-sent writes.
|
|
3. While failing over, the libxce client library shall not silently drop a request; it shall
|
|
either complete the request against a primary or surface a distinct error to the caller.
|
|
|
|
### Requirement 6: Configurable Retry Parameters and Thundering-Herd Avoidance
|
|
|
|
**Objective:** As an operator, I want retry behavior tunable with safe defaults and protection
|
|
against synchronized retries, so that failover does not overwhelm a freshly elected primary.
|
|
|
|
#### Acceptance Criteria
|
|
1. The libxce client library shall provide configurable retry parameters: maximum retries,
|
|
initial delay, maximum delay, and backoff multiplier.
|
|
2. Where no retry configuration is supplied, the libxce client library shall apply safe
|
|
production defaults.
|
|
3. When many clients fail over simultaneously, the libxce client library shall stagger their
|
|
retry attempts through randomized backoff so that they do not retry in lockstep against the new
|
|
primary.
|
|
|
|
### Requirement 7: Failover Observability
|
|
|
|
**Objective:** As an operator, I want failover activity visible in logs, so that I can confirm
|
|
and diagnose client failover during a controller outage.
|
|
|
|
#### Acceptance Criteria
|
|
1. When the libxce client library fails over to a different controller, it shall log the failover
|
|
event including the target controller endpoint.
|
|
2. When the libxce client library returns a cluster-not-ready error after exhausting its retry
|
|
budget, it shall log the exhaustion including the number of attempts made.
|
|
|
|
### Requirement 8: Redirect Responses Carry the Primary Address (used as a hint)
|
|
|
|
**Objective:** As a client application, I want a standby's redirect to *hint* where the primary is,
|
|
so libxce can usually reach it in one hop while still working correctly without the hint.
|
|
|
|
#### Acceptance Criteria
|
|
1. When a standby controller redirects a client because it knows a primary exists elsewhere, the
|
|
controller shall include the primary's host and port in the redirect response.
|
|
2. While a standby controller does not know the primary's address, the controller shall return a
|
|
no-primary status rather than a redirect that omits the address.
|
|
3. When the libxce client library receives a redirect response that includes a primary address, it
|
|
shall try that address first as a best-effort hint.
|
|
4. If the redirect carries no address, or the hinted address is unreachable or not the primary, the
|
|
libxce client library shall fall back to rotating its configured master list; correctness shall
|
|
not depend on the hint.
|