150 lines
8.8 KiB
Markdown
150 lines
8.8 KiB
Markdown
# Brief: libxce-failover
|
||
|
||
## Problem
|
||
|
||
Client applications using libxce do **not** survive controller primary failover. When the
|
||
primary controller is killed and a new one is elected, in-flight and subsequent libxce calls
|
||
fail with an opaque `Invalid argument` (EINVAL) instead of either (a) retrying transparently
|
||
against the new primary or (b) reporting a clear "cluster is not ready" during the election
|
||
window.
|
||
|
||
Reproduction (3-node HA cluster, `set masters localhost:9991,localhost:9992,localhost:9993`):
|
||
1. Primary elected; `jobsub` / `joblist` work.
|
||
2. Kill the primary (`ctrl-1`).
|
||
3. Before a new primary is elected, `joblist` should report **cluster not ready**.
|
||
4. After a new primary is elected, all commands should transparently route to the new
|
||
primary with no error.
|
||
|
||
Observed instead: `joblist` keeps returning stale cached output, then `xce_joblist: Invalid
|
||
argument` indefinitely — the client never fails over.
|
||
|
||
## Current State
|
||
|
||
- The requirement already exists: **`ha-config-bootstrap` Requirement 7 — "Client-Side
|
||
Transparent Retry for Transient Controller Errors"** (acceptance criteria 7.1–7.6), and its
|
||
tasks 10 / 10.1 / 10.2 are marked complete.
|
||
- Only the **standalone primitive** was delivered:
|
||
- `xce_request_with_retry()` is implemented at `src/lib/request.c:1956` and unit-tested in
|
||
`src/lib/testsuite/test_libxce_retry.c` (NOT_READY backoff, STANDBY+redirect reconnect,
|
||
STANDBY-without-redirect fallback, non-retriable surface-immediately, max-retries,
|
||
50-client concurrency). `xce_retry_config_t` defaults: `max_retries=20`,
|
||
`initial_retry_ms=50`, `max_retry_ms=5000`, `backoff_multiplier=2.0`.
|
||
- **Gap 1 — wiring:** every production API path bypasses the wrapper. `xce_joblist` →
|
||
`xce_get_req` (`src/lib/request.c:1373`) → plain `xce_request()` (`request.c:1385`). The
|
||
same is true for ~18 other `xce_*_req` wrappers — none call `xce_request_with_retry()`.
|
||
- **Gap 2 — multi-master discovery:** the retry primitive operates on a single `endpoint_t`
|
||
and only follows an explicit `redirect_host/redirect_port`. A *killed* primary cannot
|
||
redirect, so the client must **rotate through its configured master list** (9991/9992/9993)
|
||
to discover the new primary. That orchestration does not exist; the client connects to
|
||
`masters[0]` once in `xcecli_start()` (`src/lib/client.c:122`) and never re-selects.
|
||
- Connections are long-lived and cached in `api_ctx_t->ep`; there is no dead-primary
|
||
detection or automatic reconnect at the API level.
|
||
|
||
## Desired Outcome
|
||
|
||
Failover is transparent to client applications:
|
||
- Short failovers are hidden: libxce retries across the configured masters with backoff and
|
||
succeeds against the new primary once elected.
|
||
- If the election outlasts the retry budget, the call returns a clear `E_XCE_NOT_READY`
|
||
("cluster not ready") rather than `EINVAL`.
|
||
- Both read and write requests fail over (writes rely on existing server-side commit-barrier +
|
||
idempotency guards to dedup re-sends).
|
||
- No per-daemon ad-hoc retry loops — the behavior lives entirely in libxce.
|
||
|
||
## Approach
|
||
|
||
**Central chokepoint (Approach A).** Make failover the default behavior at the single point
|
||
all wrappers already funnel through:
|
||
|
||
1. Give the request layer access to the cluster/master list + a "current primary" index
|
||
(design choice: a back-reference on `endpoint_t`, or a small `xce_cluster_t` the endpoint
|
||
references — resolved in design phase).
|
||
2. Upgrade `xce_request_with_retry()` to, in addition to its existing redirect/backoff cases:
|
||
- rotate through the master list on connection failure (`ECONNREFUSED`/`ETIMEDOUT`/RDHUP) or
|
||
`STANDBY`-without-redirect, reconnecting to each candidate to find the new primary;
|
||
- apply the bounded backoff budget and, on expiry, surface `E_XCE_NOT_READY`.
|
||
3. Route the production `xce_*_req` wrappers through the retry/failover path (e.g. make
|
||
`xce_request()` the failover wrapper over an inner `xce_request_raw()`), wiring every API
|
||
at once.
|
||
|
||
Chosen because it is the only approach that covers the entire client surface at one
|
||
integration point, reuses the already-tested primitive, treats reads and writes uniformly, and
|
||
keeps the requirement out of caller code — matching "transparent to the client application."
|
||
|
||
## Scope
|
||
|
||
- **In:**
|
||
- Wire all production libxce request paths through transparent retry/failover.
|
||
- Multi-master rotation to discover the new primary after the connected one dies.
|
||
- Bounded-budget semantics: transparent retry, then `E_XCE_NOT_READY` ("cluster not ready").
|
||
- Failover for both read and write requests (rely on server idempotency for writes).
|
||
- Tests: extend `test_libxce_retry.c` for master-list rotation + budget→NOT_READY; a 3-node
|
||
live failover integration test (kill primary, assert NOT_READY window then transparent
|
||
recovery).
|
||
- **Out:**
|
||
- Server-side election, WAL, commit barrier, idempotency guards (already owned by the HA
|
||
server specs; this spec consumes them).
|
||
- Changing the `xce_test` REPL behavior beyond what surfaces the new error semantics.
|
||
- Per-request idempotency/retry opt-in flags (writes retry by default per decision).
|
||
|
||
## Boundary Candidates
|
||
|
||
- Master-list + current-primary selection state and rotation policy.
|
||
- The retry/failover wrapper (backoff budget, error classification, NOT_READY surfacing).
|
||
- Integration seam: how production `xce_*_req` wrappers reach the failover path.
|
||
- Error-surface mapping: connection/EINVAL vs. `E_XCE_NOT_READY` vs. `E_XCE_STANDBY` vs.
|
||
non-retriable (auth/perm/invalid/noent).
|
||
|
||
## Out of Boundary
|
||
|
||
- Any controller/server-side change (election timing, redirect content, idempotency).
|
||
- New transport/probe opcodes unless design shows rotation cannot work with existing replies.
|
||
|
||
## Upstream / Downstream
|
||
|
||
- **Upstream:** `ha-config-bootstrap` (Req 7 retry primitive + `xce_retry_config_t` + error
|
||
codes `E_XCE_NOT_READY` / `E_XCE_STANDBY` + `redirect_host/port` reply fields);
|
||
`ha-commit-barrier-rollout` (server idempotency guards that make write retries safe);
|
||
`ha-single-leader-role` / election specs (define when a standby/primary answers).
|
||
- **Downstream:** worker, scheduler, and apiserver clients — all gain transparent failover and
|
||
can drop any ad-hoc retry handling; the `/xce-deploy-ha` live test harness.
|
||
|
||
## Existing Spec Touchpoints
|
||
|
||
- **Extends:** consumes and completes the intent of `ha-config-bootstrap` Req 7 (the primitive
|
||
was delivered but never integrated); this spec owns the integration + multi-master failover.
|
||
- **Adjacent:** `ha-repl-hardening`, `ha-convergence-recovery` (server-side failover behavior);
|
||
`worker-rejoin-resync` (worker-side rejoin, distinct from client-library failover).
|
||
|
||
## Protocol Correction (2026-06-16, requirements phase)
|
||
|
||
The discovery notes above assumed the controller signals non-primary via `E_XCE_STANDBY` with
|
||
`redirect_host/redirect_port` fields. Code review found the real taxonomy (`errnox.h:22-25`):
|
||
`E_XCE_REDIRECT` (standby knows a primary exists; `join.c:260-268`), `E_XCE_NOMASTER` (standby,
|
||
no primary known yet; `join.c:270-278`), `E_XCE_NOT_READY` (INIT/ELECTING), `E_XCE_STANDBY`
|
||
(primary-only op on a standby). The controller's redirect path sends `E_XCE_REDIRECT` with
|
||
`data_which = nodata` — **status only, no address** — so the existing `xce_request_with_retry()`
|
||
condition (`E_XCE_STANDBY` + populated redirect fields, `request.c:2010`) never matches a real
|
||
controller. Decisions (confirmed): classify on the actual statuses — `E_XCE_REDIRECT` →
|
||
fast-rotate the master list to find the existing primary (use an address if the reply carries
|
||
one, else rotate); `E_XCE_NOMASTER`/`E_XCE_NOT_READY` → backoff → cluster-not-ready;
|
||
`E_XCE_STANDBY` → rotate. The existing retry unit tests (keyed on the non-emitted shape) are
|
||
corrected to the real statuses. Tracked as bug-608.
|
||
|
||
**Scope expansion (confirmed):** the redirect address is absent today only because `join.c`
|
||
never fills the reply's `redirect_host/redirect_port` fields — those fields already exist in the
|
||
wire schema (`controller.capnp:568-569`) and the reply struct (`typemap.h:225-226`), and the
|
||
controller already knows the address (`join.c:263`). One small controller change to populate them
|
||
is now **in scope** (Requirement 8), making redirect-based failover deterministic (direct connect
|
||
to the primary, O(1)); client-side master-list rotation remains as a defensive fallback for an
|
||
older controller that omits the address.
|
||
|
||
## Constraints
|
||
|
||
- C library (libxce); follow project error-handling (`errnox`, early-return) and async-I/O
|
||
rules — `xce_request`/`xce_reply` non-blocking; never block the iomgr thread.
|
||
- TDD mandatory: failing tests in `src/lib/testsuite/` before implementation.
|
||
- Touches HA/failover behavior → full Kiro workflow (requirements → design → tasks), not the
|
||
quick path, per CLAUDE.md.
|
||
- Reuse the existing `xce_request_with_retry()` primitive and its config struct; do not fork a
|
||
second retry implementation.
|