myxce/.kiro/specs/component-failover-adoption/requirements.md

Requirements Document

Project Description (Input)

component-failover-adoption — pre-scoped roadmap backlog item (roadmap.md:64, recorded 2026-06-16, HARD-dependent on libxce-failover, which shipped GO 2026-07-18 — this item is now unblocked). Make the C daemons (scheduler, worker) and the Go apiserver actually exercise libxce's transparent multi-master failover, matching xce_test's masters-setting behavior. Two roadmap-defined integration items:

1. Each component must hand libxce the full master/candidate list, not a single endpoint (today even the CLI uses only conf->masters[0]).
2. Bounded retry makes the already-blocking xce_request block longer (seconds of backoff) — for the async C daemons this must be verified liveness-safe: it must never stall a thread that also drives heartbeats (the project-wide "never block iomgr/delaymgr" thread-role contract, ha-async-io.md).

Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for scheduler, worker, apiserver. xce_test is explicitly OUT of scope — it is the already-correct in-spec verification client, not a target.

Background (verification findings, 2026-07-19)

xce_test's masters setting (src/controller/testsuite/test.c:345-377) is the reference implementation: it parses a host:port,... list via xce_parse_peers(), sets config.nmasters = candidates_count, fills the full config.masters[] array, and calls xcecli_ctx_init() once — so libxce's own client_failover.c EndpointRotator handles mid-session failover across all candidates.

None of the three production daemons match this pattern today; each has a different gap:

• worker (xcecd): --peers is parsed into cluster_cfg, but cc_ctx->cluster_cfg is only assigned after xcecd_start() returns (src/worker/src/main.c:610), while xcecd_start() already performs the first join via init_client_config() (src/worker/src/controller_connector.c:127-158, hardcoded nmasters=1) — so the initial connection ignores --peers entirely and only later reconnects (checking_connection() → xcecd_join_cluster(), controller_connector.c:237-293) use the candidate list, one candidate at a time (app-level retry, not lib-level failover).
• scheduler (xce_scheduler): --peers candidate loop is correctly wired before the initial join (src/scheduler/src/main.c:124-156, scheduler.c:462-495), but there is no runtime reconnect/rejoin logic anywhere in the scheduler — if the joined controller later dies, the scheduler has no path to another candidate.
• apiserver: backend.peers (YAML) is tried once at startup via a Go-level loop (src/frontend/apiserver/backend/clib.go:116-133, RegisterToBackend()), each attempt using nmasters=1. No periodic/background reconnect across peers exists — HA coverage is startup-only.

In all three daemons, xce_client_config_t.nmasters is always set to 1 per connection attempt, so libxce's own failover rotator (client_failover.c) never actually engages for them — each daemon hand-rolls its own (incomplete) outer retry loop instead.

A separate, partially-built shared primitive already exists in libxce (src/include/xce/cluster.h: xce_connect_cluster(), xce_reconnect_init/step/fini(), backed by xce_parse_peers()/xce_cluster_config_validate()) — but its production connect path (do_connect() in src/lib/cluster.c) is a stub that always returns NULL outside unit tests, and it operates on a raw endpoint_t*, not the higher-level xcecli_ctx_t the daemons actually use for the join/request/reply protocol. No production code calls it. Whether the design phase completes/adopts this primitive or builds directly on xce_client_config_t.nmasters (the libxce-failover/client_failover.c path xce_test already uses) is a design decision, not a requirements one — both existing mechanisms are noted here as prior art this feature must reconcile with, not duplicate.

Background (verification findings, 2026-07-19)

xce_test's masters setting (src/controller/testsuite/test.c:345-377) is the reference implementation: it parses a host:port,... list via xce_parse_peers(), sets config.nmasters = candidates_count, fills the full config.masters[] array, and calls xcecli_ctx_init() once — so libxce's own client_failover.c EndpointRotator handles mid-session failover across all candidates.

None of the three production daemons match this pattern; each has a different gap:

• worker (xcecd): --peers is parsed into cluster_cfg, but cc_ctx->cluster_cfg is only assigned after xcecd_start() returns (src/worker/src/main.c:610), while xcecd_start() already performs the first join via init_client_config() (src/worker/src/controller_connector.c:127-158, hardcoded nmasters=1) — so the initial connection ignores --peers entirely and only later reconnects (checking_connection() → xcecd_join_cluster(), controller_connector.c:237-293) use the candidate list, one candidate at a time (app-level retry, not lib-level failover).
• scheduler (xce_scheduler): --peers candidate loop is correctly wired before the initial join (src/scheduler/src/main.c:124-156, scheduler.c:462-495), but there is no runtime reconnect/rejoin logic anywhere in the scheduler — if the joined controller later dies, the scheduler has no path to another candidate.
• apiserver: backend.peers (YAML) is tried once at startup via a Go-level loop (src/frontend/apiserver/backend/clib.go:116-133, RegisterToBackend()), each attempt using nmasters=1. No periodic/background reconnect across peers exists — HA coverage is startup-only.

In all three daemons, xce_client_config_t.nmasters is always set to 1 per connection attempt, so libxce's own failover rotator (client_failover.c) never actually engages for them — each daemon hand-rolls its own (incomplete) outer retry loop instead.

Boundary Context

• In scope: scheduler, worker, and apiserver each using their full configured controller-candidate list (not just the first entry) for both the initial connection and any later reconnect; automatic reconnect across candidates for all three components (today scheduler and apiserver have none); liveness-safe retry/backoff on the worker and scheduler so a slow/unreachable candidate never stalls their other liveness-critical duties; consistent candidate-list configuration syntax across all three components; operator-visible diagnostics for connect/reconnect attempts and failures; apiserver behavior toward its own API clients while its backend connection is down or being re-established.
• Out of scope: xce_test (already the correct reference client, not a target of this feature); raising the validated 3-candidate limit; controller-side changes (peer authentication, validate_host(), redirect-reply population — already shipped by libxce-failover); TLS/certificate distribution mechanics beyond the existing CA-cert configuration; the lightweight session-based rejoin/re-attach optimization (separate backlog item); recovering worker job-state that was lost during a disconnect window (owned by the separate worker-rejoin-resync item — this feature restores the connection, not job-state content).
• Adjacent expectations: depends on libxce-failover (shipped) for the underlying retry primitive and controller status taxonomy (redirect vs. standby vs. not-ready); does not duplicate or replace that mechanism, only ensures each component actually drives it with a complete candidate list.

Requirements

Requirement 1: Full candidate list used on initial connection

Objective: As an HA cluster operator, I want scheduler, worker, and apiserver to attempt their initial controller connection using the complete configured candidate list, so that startup succeeds even when the first-listed candidate is not the current primary.

Acceptance Criteria

1. When a component is started with more than one controller candidate configured, the component shall attempt connection to each configured candidate, in configured order, until one accepts the join as primary.
2. If the first attempted candidate is unreachable, the component shall proceed to the next configured candidate without operator intervention or a restart.
3. If a configured candidate responds as standby rather than primary, the component shall treat that as informational and proceed to the next configured candidate, not as a fatal connection error.
4. The worker component shall use the fully configured candidate list on its first connection attempt at startup, not only on subsequent reconnects.
5. Where only one candidate (or none) is configured, the component shall behave as it does today in single-controller mode.
6. If every configured candidate is unreachable or rejects the join at startup, then the component shall report a clear startup failure identifying which candidates were attempted.

Requirement 2: Automatic reconnect after controller failover

Objective: As an HA cluster operator, I want scheduler, worker, and apiserver to automatically reconnect to a surviving controller candidate when their active controller connection is lost, so that a controller failover does not require manually restarting these daemons.

Acceptance Criteria

1. When a component's active connection to its controller is lost, the component shall attempt to reconnect using the complete configured candidate list, not only the previously connected candidate.
2. While reconnecting, the component shall keep retrying across the full configured candidate list until one candidate accepts the connection as primary or the component is shut down by an operator.
3. If a reconnect sweep exhausts all configured candidates without success, then the component shall continue retrying rather than permanently giving up, so the component self-heals once a primary becomes reachable again.
4. When apiserver reconnects to a new primary after a controller failover, apiserver shall resume receiving backend push notifications without requiring an apiserver process restart.

Requirement 3: Reconnect retry does not block other liveness-critical work

Objective: As an HA cluster operator, I want the worker and scheduler's controller-reconnect retries to never block their other liveness-critical processing, so that a slow or unreachable controller candidate does not make the daemon itself appear unresponsive.

Acceptance Criteria

1. While a worker or scheduler is retrying a controller connection (including any backoff delay between attempts), the component shall continue performing its other liveness-critical duties (worker: hardware/process monitoring and job-state tracking; scheduler: its allocation/decision processing) without being blocked by the retry attempt or backoff delay.
2. The worker and scheduler's reconnect retries shall not delay or prevent the component's own heartbeat responses to a connected controller, so a stalled reconnect attempt elsewhere cannot itself trigger an unnecessary failover.

Requirement 4: apiserver behavior toward API clients during backend disconnection

Objective: As an API client of apiserver, I want a clear, immediate response when the backend controller connection is down, so that I am not left waiting indefinitely for requests that cannot currently be served.

Acceptance Criteria

1. When apiserver's connection to the controller is down and a reconnect is in progress, apiserver shall respond to API requests that require the backend with a clear service-unavailable style error rather than hanging until a connection is restored.
2. While the backend reconnect is in progress, apiserver shall continue attempting to serve API requests that do not require the backend connection.

Requirement 5: Operator-visible connect/reconnect diagnostics

Objective: As an HA cluster operator, I want visibility into which controller candidates each component tried and why each attempt failed, so that I can distinguish a real outage from a configuration mistake.

Acceptance Criteria

1. When a component attempts a connection or reconnection to a controller candidate, the component shall log the candidate attempted and the outcome (success, unreachable, or standby-redirect).
2. If a component exhausts all configured candidates during a connection or reconnection attempt, then the component shall log a summary identifying every candidate attempted and the reason each failed.

Requirement 6: Consistent candidate-list configuration across components

Objective: As an HA cluster operator, I want to configure the controller candidate list the same way for scheduler, worker, and apiserver, so that I do not need to learn or maintain three different configuration formats for the same setting.

Acceptance Criteria

1. The candidate-list configuration accepted by apiserver shall use the same [name=]host:port[,...] syntax already accepted by scheduler and worker.
2. If a component's candidate-list configuration is malformed, then the component shall reject it with an error identifying the malformed entry, consistent with how the other components report malformed candidate-list configuration.