lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/libxce-failover/requirements.md

12 KiB
Raw Blame History

Requirements Document

Introduction

This feature makes XCE controller primary failover transparent to client applications built on libxce (the XCE client library), including the worker, scheduler, and API server daemons and the xce_test REPL.

Today, when the connected controller primary is killed in an HA cluster, libxce keeps using the dead connection and returns an opaque invalid-argument error indefinitely: it never fails over to the newly elected primary and never reports a clear "cluster not ready" during the election window. The upstream requirement (ha-config-bootstrap Requirement 7) delivered only a standalone retry primitive that no production request path invokes and that cannot discover a new primary after the connected one dies. This spec completes that intent on the client side: it routes all client requests through transparent retry/failover, rotates the configured master list to locate the new primary, and surfaces a distinct cluster-not-ready error only when an election outlasts the retry budget.

Boundary Context

  • In scope: client-side transparent failover for all libxce request types; discovery of the new primary by rotating the configured master list, using the redirect address (when present) as a best-effort hint to try first; bounded transparent retry that surfaces a distinct cluster-not-ready error on budget exhaustion; failover for both read and write requests; configurable retry/backoff parameters with safe defaults; operator-visible logging of failover events; one small controller change so that redirect responses carry the primary's host and port (the wire schema and reply struct already define these fields); end-to-end verification with the xce_test client (the controller test harness, and the original reproduction case for this bug).
  • Out of scope: controller-side election, log replication, commit barrier, and write idempotency (consumed, not changed); introducing new probe or transport message types or new reply fields (the redirect address fields already exist); per-request retry opt-in flags (writes fail over by default); adoption by the worker, scheduler, and apiserver daemons — their master-list configuration and liveness-safe handling of bounded blocking are owned by the follow-on spec component-failover-adoption (depends on this spec).
  • Adjacent expectations: libxce classifies controller responses by the distinct statuses the controller actually emits — a redirect status (a primary exists but this node is a standby), a no-primary status (this standby knows no primary yet), a not-ready status (the controller is still initializing or electing), and a standby status (the operation is primary-only). With the in-scope controller change, a redirect response carries the primary's host and port; libxce treats this as a best-effort hint — it tries the hinted address first, and rotates its configured master list when the hint is absent, stale, or unreachable. Correctness never depends on the hint. libxce relies on controllers to deduplicate re-sent writes through their commit-barrier idempotency guards, and on the configured master list (e.g. set masters host:port,host:port,host:port) being the full set of controller candidates.
  • Stateful consumers: a pure client (xce_test) needs only reconnect + resend. Stateful daemons (apiserver, worker, scheduler) hold a session established via join; after a failover they must re-establish it on the new primary (and a worker re-syncs its running jobs). libxce provides a reconnect/rejoin hook for this; the hook content is owned by component-failover-adoption and worker-rejoin-resync. Fire-and-forget notifications have no reply, so libxce cannot detect a standby for them or guarantee delivery across a failover — redelivery is worker-rejoin-resync.

Requirements

Requirement 1: Transparent Failover for All Client Requests

Objective: As a developer of a libxce-based daemon, I want every controller request to fail over automatically inside the library, so that I do not write ad-hoc retry or reconnect loops in caller code.

Acceptance Criteria

  1. The libxce client library shall route every controller request, both read/query and mutating/write requests, through the transparent retry-and-failover path.
  2. When controller failover occurs and a new primary becomes available, the libxce client library shall complete the next request against the new primary without requiring any change to caller code.
  3. While failover handling is active, the libxce client library shall preserve the existing public API call signatures and success/error return conventions so that existing callers continue to compile and run unchanged.

Requirement 2: New-Primary Discovery Across the Configured Master List

Objective: As a client application, I want libxce to find the new primary after the connected one dies, so that my requests continue to reach an authoritative controller.

Acceptance Criteria

  1. If the connected controller becomes unreachable through connection refused, connection timeout, or peer reset, the libxce client library shall attempt the other controllers in its configured master list to locate the current primary.
  2. When a contacted controller responds with a redirect status indicating that a primary exists elsewhere, the libxce client library shall locate the primary by rotating its configured master list, trying the redirect address (when the response carries one) first as a best-effort hint.
  3. When a contacted controller responds that the requested operation is primary-only (a standby status), the libxce client library shall treat it as a redirect and attempt the other controllers in its configured master list.
  4. When a controller accepts and serves the request as primary, the libxce client library shall treat it as the current primary and direct subsequent requests to it.
  5. The libxce client library shall not consider a request successful when it was answered by a controller reporting a redirect, standby, no-primary, or not-ready status.

Requirement 3: Election-Window Behavior and Cluster-Not-Ready Reporting

Objective: As a client application, I want short failovers hidden and longer outages reported clearly, so that I can distinguish "retry later" from a real failure.

Acceptance Criteria

  1. While every reachable controller in the configured master list reports no primary exists (a no-primary or not-ready status), the libxce client library shall retry across the configured controllers with exponential backoff until the retry budget is exhausted.
  2. When a primary becomes available before the retry budget is exhausted, the libxce client library shall complete the request successfully and return no error to the caller.
  3. If the retry budget is exhausted while no controller has become primary, the libxce client library shall return a distinct cluster-not-ready error to the caller.
  4. The cluster-not-ready error shall be distinguishable by the caller from transport connection errors and from request-validation errors, so that a no-primary window is never reported as an opaque invalid-argument error.

Requirement 4: Error Classification — Retriable vs. Terminal

Objective: As a client application, I want transient controller errors retried and permanent errors surfaced immediately, so that I neither hang on unrecoverable errors nor fail on recoverable ones.

Acceptance Criteria

  1. If a controller returns a no-primary or not-ready status, the libxce client library shall retry the request with exponential backoff.
  2. If a controller returns a redirect status (a primary exists elsewhere), the libxce client library shall locate and switch to the primary per Requirement 2 and retry promptly, without waiting for the full backoff delay used for the no-primary case.
  3. If a controller returns a standby status (the operation is primary-only), the libxce client library shall switch to another configured controller per Requirement 2 and retry.
  4. If a transport-level connection refused, timeout, or peer reset occurs, the libxce client library shall reconnect or rotate to another configured controller and retry.
  5. If a controller returns a non-retriable error, specifically authentication failure, permission denied, invalid request, or not found, the libxce client library shall surface that error to the caller immediately without retry.

Requirement 5: Read and Write Failover Semantics

Objective: As a client application, I want both queries and mutating operations to survive failover without duplicating side effects, so that failover is safe as well as transparent.

Acceptance Criteria

  1. The libxce client library shall apply transparent failover to both read/query requests and mutating/write requests, such as job submission.
  2. When a write request is retried after a failover, the libxce client library shall not cause the operation to be applied more than once, relying on the controller's idempotency handling for re-sent writes.
  3. While failing over, the libxce client library shall not silently drop a request; it shall either complete the request against a primary or surface a distinct error to the caller.

Requirement 6: Configurable Retry Parameters and Thundering-Herd Avoidance

Objective: As an operator, I want retry behavior tunable with safe defaults and protection against synchronized retries, so that failover does not overwhelm a freshly elected primary.

Acceptance Criteria

  1. The libxce client library shall provide configurable retry parameters: maximum retries, initial delay, maximum delay, and backoff multiplier.
  2. Where no retry configuration is supplied, the libxce client library shall apply safe production defaults.
  3. When many clients fail over simultaneously, the libxce client library shall stagger their retry attempts through randomized backoff so that they do not retry in lockstep against the new primary.

Requirement 7: Failover Observability

Objective: As an operator, I want failover activity visible in logs, so that I can confirm and diagnose client failover during a controller outage.

Acceptance Criteria

  1. When the libxce client library fails over to a different controller, it shall log the failover event including the target controller endpoint.
  2. When the libxce client library returns a cluster-not-ready error after exhausting its retry budget, it shall log the exhaustion including the number of attempts made.

Requirement 8: Redirect Responses Carry the Primary Address (used as a hint)

Objective: As a client application, I want a standby's redirect to hint where the primary is, so libxce can usually reach it in one hop while still working correctly without the hint.

Acceptance Criteria

  1. When a standby controller redirects a client because it knows a primary exists elsewhere, the controller shall include the primary's host and port in the redirect response.
  2. While a standby controller does not know the primary's address, the controller shall return a no-primary status rather than a redirect that omits the address.
  3. When the libxce client library receives a redirect response that includes a primary address, it shall try that address first as a best-effort hint.
  4. If the redirect carries no address, or the hinted address is unreachable or not the primary, the libxce client library shall fall back to rotating its configured master list; correctness shall not depend on the hint.