90 lines
13 KiB
Markdown
90 lines
13 KiB
Markdown
# Requirements Document
|
|
|
|
## Project Description (Input)
|
|
`component-failover-adoption` — pre-scoped roadmap backlog item (`roadmap.md:64`, recorded 2026-06-16, HARD-dependent on `libxce-failover`, which shipped GO 2026-07-18 — this item is now unblocked). Make the C daemons (scheduler, worker) and the Go apiserver actually exercise libxce's transparent multi-master failover, matching xce_test's `masters`-setting behavior. Two roadmap-defined integration items:
|
|
1. Each component must hand libxce the **full master/candidate list**, not a single endpoint (today even the CLI uses only `conf->masters[0]`).
|
|
2. Bounded retry makes the already-blocking `xce_request` block **longer** (seconds of backoff) — for the async C daemons this must be verified **liveness-safe**: it must never stall a thread that also drives heartbeats (the project-wide "never block iomgr/delaymgr" thread-role contract, `ha-async-io.md`).
|
|
|
|
Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for **scheduler, worker, apiserver**. `xce_test` is explicitly OUT of scope — it is the already-correct in-spec verification client, not a target.
|
|
|
|
### Background (verification findings, 2026-07-19)
|
|
|
|
`xce_test`'s `masters` setting (`src/controller/testsuite/test.c:345-377`) is the reference implementation: it parses a `host:port,...` list via `xce_parse_peers()`, sets `config.nmasters = candidates_count`, fills the full `config.masters[]` array, and calls `xcecli_ctx_init()` once — so libxce's own `client_failover.c` EndpointRotator handles mid-session failover across all candidates.
|
|
|
|
None of the three production daemons match this pattern today; each has a different gap:
|
|
|
|
- **worker (xcecd)**: `--peers` is parsed into `cluster_cfg`, but `cc_ctx->cluster_cfg` is only assigned *after* `xcecd_start()` returns (`src/worker/src/main.c:610`), while `xcecd_start()` already performs the first join via `init_client_config()` (`src/worker/src/controller_connector.c:127-158`, hardcoded `nmasters=1`) — so the initial connection ignores `--peers` entirely and only later reconnects (`checking_connection()` → `xcecd_join_cluster()`, `controller_connector.c:237-293`) use the candidate list, one candidate at a time (app-level retry, not lib-level failover).
|
|
- **scheduler (xce_scheduler)**: `--peers` candidate loop is correctly wired before the initial join (`src/scheduler/src/main.c:124-156`, `scheduler.c:462-495`), but there is no runtime reconnect/rejoin logic anywhere in the scheduler — if the joined controller later dies, the scheduler has no path to another candidate.
|
|
- **apiserver**: `backend.peers` (YAML) is tried once at startup via a Go-level loop (`src/frontend/apiserver/backend/clib.go:116-133`, `RegisterToBackend()`), each attempt using `nmasters=1`. No periodic/background reconnect across peers exists — HA coverage is startup-only.
|
|
|
|
In all three daemons, `xce_client_config_t.nmasters` is always set to `1` per connection attempt, so libxce's own failover rotator (`client_failover.c`) never actually engages for them — each daemon hand-rolls its own (incomplete) outer retry loop instead.
|
|
|
|
A separate, partially-built shared primitive already exists in libxce (`src/include/xce/cluster.h`: `xce_connect_cluster()`, `xce_reconnect_init/step/fini()`, backed by `xce_parse_peers()`/`xce_cluster_config_validate()`) — but its production connect path (`do_connect()` in `src/lib/cluster.c`) is a stub that always returns `NULL` outside unit tests, and it operates on a raw `endpoint_t*`, not the higher-level `xcecli_ctx_t` the daemons actually use for the join/request/reply protocol. No production code calls it. Whether the design phase completes/adopts this primitive or builds directly on `xce_client_config_t.nmasters` (the `libxce-failover`/`client_failover.c` path xce_test already uses) is a design decision, not a requirements one — both existing mechanisms are noted here as prior art this feature must reconcile with, not duplicate.
|
|
|
|
### Background (verification findings, 2026-07-19)
|
|
|
|
`xce_test`'s `masters` setting (`src/controller/testsuite/test.c:345-377`) is the reference implementation: it parses a `host:port,...` list via `xce_parse_peers()`, sets `config.nmasters = candidates_count`, fills the full `config.masters[]` array, and calls `xcecli_ctx_init()` once — so libxce's own `client_failover.c` EndpointRotator handles mid-session failover across all candidates.
|
|
|
|
None of the three production daemons match this pattern; each has a different gap:
|
|
|
|
- **worker (xcecd)**: `--peers` is parsed into `cluster_cfg`, but `cc_ctx->cluster_cfg` is only assigned *after* `xcecd_start()` returns (`src/worker/src/main.c:610`), while `xcecd_start()` already performs the first join via `init_client_config()` (`src/worker/src/controller_connector.c:127-158`, hardcoded `nmasters=1`) — so the initial connection ignores `--peers` entirely and only later reconnects (`checking_connection()` → `xcecd_join_cluster()`, `controller_connector.c:237-293`) use the candidate list, one candidate at a time (app-level retry, not lib-level failover).
|
|
- **scheduler (xce_scheduler)**: `--peers` candidate loop is correctly wired before the initial join (`src/scheduler/src/main.c:124-156`, `scheduler.c:462-495`), but there is no runtime reconnect/rejoin logic anywhere in the scheduler — if the joined controller later dies, the scheduler has no path to another candidate.
|
|
- **apiserver**: `backend.peers` (YAML) is tried once at startup via a Go-level loop (`src/frontend/apiserver/backend/clib.go:116-133`, `RegisterToBackend()`), each attempt using `nmasters=1`. No periodic/background reconnect across peers exists — HA coverage is startup-only.
|
|
|
|
In all three daemons, `xce_client_config_t.nmasters` is always set to `1` per connection attempt, so libxce's own failover rotator (`client_failover.c`) never actually engages for them — each daemon hand-rolls its own (incomplete) outer retry loop instead.
|
|
|
|
## Boundary Context
|
|
|
|
- **In scope**: scheduler, worker, and apiserver each using their *full* configured controller-candidate list (not just the first entry) for both the initial connection and any later reconnect; automatic reconnect across candidates for all three components (today scheduler and apiserver have none); liveness-safe retry/backoff on the worker and scheduler so a slow/unreachable candidate never stalls their other liveness-critical duties; consistent candidate-list configuration syntax across all three components; operator-visible diagnostics for connect/reconnect attempts and failures; apiserver behavior toward its own API clients while its backend connection is down or being re-established.
|
|
- **Out of scope**: `xce_test` (already the correct reference client, not a target of this feature); raising the validated 3-candidate limit; controller-side changes (peer authentication, `validate_host()`, redirect-reply population — already shipped by `libxce-failover`); TLS/certificate distribution mechanics beyond the existing CA-cert configuration; the lightweight session-based rejoin/re-attach optimization (separate backlog item); recovering worker job-state that was lost during a disconnect window (owned by the separate `worker-rejoin-resync` item — this feature restores the *connection*, not job-state content).
|
|
- **Adjacent expectations**: depends on `libxce-failover` (shipped) for the underlying retry primitive and controller status taxonomy (redirect vs. standby vs. not-ready); does not duplicate or replace that mechanism, only ensures each component actually drives it with a complete candidate list.
|
|
|
|
## Requirements
|
|
|
|
### Requirement 1: Full candidate list used on initial connection
|
|
**Objective:** As an HA cluster operator, I want scheduler, worker, and apiserver to attempt their initial controller connection using the complete configured candidate list, so that startup succeeds even when the first-listed candidate is not the current primary.
|
|
|
|
#### Acceptance Criteria
|
|
1. When a component is started with more than one controller candidate configured, the component shall attempt connection to each configured candidate, in configured order, until one accepts the join as primary.
|
|
2. If the first attempted candidate is unreachable, the component shall proceed to the next configured candidate without operator intervention or a restart.
|
|
3. If a configured candidate responds as standby rather than primary, the component shall treat that as informational and proceed to the next configured candidate, not as a fatal connection error.
|
|
4. The worker component shall use the fully configured candidate list on its first connection attempt at startup, not only on subsequent reconnects.
|
|
5. Where only one candidate (or none) is configured, the component shall behave as it does today in single-controller mode.
|
|
6. If every configured candidate is unreachable or rejects the join at startup, then the component shall report a clear startup failure identifying which candidates were attempted.
|
|
|
|
### Requirement 2: Automatic reconnect after controller failover
|
|
**Objective:** As an HA cluster operator, I want scheduler, worker, and apiserver to automatically reconnect to a surviving controller candidate when their active controller connection is lost, so that a controller failover does not require manually restarting these daemons.
|
|
|
|
#### Acceptance Criteria
|
|
1. When a component's active connection to its controller is lost, the component shall attempt to reconnect using the complete configured candidate list, not only the previously connected candidate.
|
|
2. While reconnecting, the component shall keep retrying across the full configured candidate list until one candidate accepts the connection as primary or the component is shut down by an operator.
|
|
3. If a reconnect sweep exhausts all configured candidates without success, then the component shall continue retrying rather than permanently giving up, so the component self-heals once a primary becomes reachable again.
|
|
4. When apiserver reconnects to a new primary after a controller failover, apiserver shall resume receiving backend push notifications without requiring an apiserver process restart.
|
|
|
|
### Requirement 3: Reconnect retry does not block other liveness-critical work
|
|
**Objective:** As an HA cluster operator, I want the worker and scheduler's controller-reconnect retries to never block their other liveness-critical processing, so that a slow or unreachable controller candidate does not make the daemon itself appear unresponsive.
|
|
|
|
#### Acceptance Criteria
|
|
1. While a worker or scheduler is retrying a controller connection (including any backoff delay between attempts), the component shall continue performing its other liveness-critical duties (worker: hardware/process monitoring and job-state tracking; scheduler: its allocation/decision processing) without being blocked by the retry attempt or backoff delay.
|
|
2. The worker and scheduler's reconnect retries shall not delay or prevent the component's own heartbeat responses to a connected controller, so a stalled reconnect attempt elsewhere cannot itself trigger an unnecessary failover.
|
|
|
|
### Requirement 4: apiserver behavior toward API clients during backend disconnection
|
|
**Objective:** As an API client of apiserver, I want a clear, immediate response when the backend controller connection is down, so that I am not left waiting indefinitely for requests that cannot currently be served.
|
|
|
|
#### Acceptance Criteria
|
|
1. When apiserver's connection to the controller is down and a reconnect is in progress, apiserver shall respond to API requests that require the backend with a clear service-unavailable style error rather than hanging until a connection is restored.
|
|
2. While the backend reconnect is in progress, apiserver shall continue attempting to serve API requests that do not require the backend connection.
|
|
|
|
### Requirement 5: Operator-visible connect/reconnect diagnostics
|
|
**Objective:** As an HA cluster operator, I want visibility into which controller candidates each component tried and why each attempt failed, so that I can distinguish a real outage from a configuration mistake.
|
|
|
|
#### Acceptance Criteria
|
|
1. When a component attempts a connection or reconnection to a controller candidate, the component shall log the candidate attempted and the outcome (success, unreachable, or standby-redirect).
|
|
2. If a component exhausts all configured candidates during a connection or reconnection attempt, then the component shall log a summary identifying every candidate attempted and the reason each failed.
|
|
|
|
### Requirement 6: Consistent candidate-list configuration across components
|
|
**Objective:** As an HA cluster operator, I want to configure the controller candidate list the same way for scheduler, worker, and apiserver, so that I do not need to learn or maintain three different configuration formats for the same setting.
|
|
|
|
#### Acceptance Criteria
|
|
1. The candidate-list configuration accepted by apiserver shall use the same `[name=]host:port[,...]` syntax already accepted by scheduler and worker.
|
|
2. If a component's candidate-list configuration is malformed, then the component shall reject it with an error identifying the malformed entry, consistent with how the other components report malformed candidate-list configuration.
|