myxce/.kiro/specs/component-failover-adoption/requirements.md

Requirements Document

Project Description (Input)

component-failover-adoption — pre-scoped roadmap backlog item (roadmap.md:64, recorded 2026-06-16, HARD-dependent on libxce-failover, which shipped GO 2026-07-18 — this item is now unblocked). Make the C daemons (scheduler, worker) and the Go apiserver actually exercise libxce's transparent multi-master failover, matching xce_test's masters-setting behavior. Two roadmap-defined integration items:

1. Each component must hand libxce the full master/candidate list, not a single endpoint (today even the CLI uses only conf->masters[0]).
2. Bounded retry makes the already-blocking xce_request block longer (seconds of backoff) — for the async C daemons this must be verified liveness-safe: it must never stall a thread that also drives heartbeats (the project-wide "never block iomgr/delaymgr" thread-role contract, ha-async-io.md).

Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for scheduler, worker, apiserver. xce_test is explicitly OUT of scope — it is the already-correct in-spec verification client, not a target.

Background (verification findings, 2026-07-19)

xce_test's masters setting (src/controller/testsuite/test.c:345-377) is the reference implementation: it parses a host:port,... list via xce_parse_peers(), sets config.nmasters = candidates_count, fills the full config.masters[] array, and calls xcecli_ctx_init() once — so libxce's own client_failover.c EndpointRotator handles mid-session failover across all candidates.

None of the three production daemons match this pattern today; each has a different gap:

• worker (xcecd): --peers is parsed into cluster_cfg, but cc_ctx->cluster_cfg is only assigned after xcecd_start() returns (src/worker/src/main.c:537), while xcecd_start() already performs the first join via init_client_config() (src/worker/src/controller_connector.c:129-160, hardcoded nmasters=1) — so the initial connection ignores --peers entirely and only later reconnects (checking_connection() → xcecd_join_cluster(), controller_connector.c:239-312) use the candidate list, one candidate at a time (app-level retry, not lib-level failover).
• scheduler (xce_scheduler): --peers candidate loop is correctly wired before the initial join (src/scheduler/src/main.c:299-324, scheduler.c:564-624), but there is no runtime reconnect/rejoin logic anywhere in the scheduler — if the joined controller later dies, the scheduler has no path to another candidate.
• apiserver: backend.peers (YAML) is tried once at startup via a Go-level loop (src/frontend/apiserver/backend/clib.go:116-133, RegisterToBackend()), each attempt using nmasters=1. No periodic/background reconnect across peers exists — HA coverage is startup-only.

In all three daemons, xce_client_config_t.nmasters is always set to 1 per connection attempt, so libxce's own failover rotator (client_failover.c) never actually engages for them — each daemon hand-rolls its own (incomplete) outer retry loop instead.

Boundary Context

• In scope: scheduler, worker, and apiserver each using their full configured controller-candidate list (not just the first entry) for both the initial connection and any later reconnect; automatic reconnect across candidates for all three components (today scheduler and apiserver have none); liveness-safe retry/backoff on the worker and scheduler so a slow/unreachable candidate never stalls their other liveness-critical duties; consistent candidate-list configuration syntax across all three components; operator-visible diagnostics for connect/reconnect attempts and failures; apiserver behavior toward its own API clients while its backend connection is down or being re-established.
• Out of scope: xce_test (already the correct reference client, not a target of this feature); controller-side changes (peer authentication, validate_host(), redirect-reply population — already shipped by libxce-failover); TLS/certificate distribution mechanics beyond the existing CA-cert configuration; the lightweight session-based rejoin/re-attach optimization (separate backlog item); recovering worker job-state that was lost during a disconnect window (owned by the separate worker-rejoin-resync item — this feature restores the connection, not job-state content). Reconciling the candidate-count limit (N_MAX_MASTERS deriving from config.h's MAX_HA_CANDIDATES, Req 1 AC7) is now explicitly in scope — see Requirement 1; an earlier draft of this document listed raising it as out of scope, superseded 2026-08-10.
• Adjacent expectations: depends on libxce-failover (shipped) for the underlying retry primitive and controller status taxonomy (redirect vs. standby vs. not-ready); does not duplicate or replace that mechanism, only ensures each component actually drives it with a complete candidate list.

Requirements

Requirement 1: Full candidate list used on initial connection

Objective: As an HA cluster operator, I want scheduler, worker, and apiserver to attempt their initial controller connection using the complete configured candidate list, so that startup succeeds even when the first-listed candidate is not the current primary.

Acceptance Criteria

1. When a component is started with more than one controller candidate configured, the component shall attempt connection to each configured candidate, in configured order, until one accepts the join as primary.
2. If the first attempted candidate is unreachable, the component shall proceed to the next configured candidate without operator intervention or a restart.
3. If a configured candidate responds as standby rather than primary, the component shall treat that as informational and proceed to the next configured candidate, not as a fatal connection error.
4. The worker component shall use the fully configured candidate list on its first connection attempt at startup, not only on subsequent reconnects.
5. Where only one candidate is configured — whether from a single-entry --peers/backend.peers list, or synthesized from the legacy single-host default when no candidate list is given at all — the component shall treat it as a one-element candidate list and drive it through the exact same validate/convert/connect pipeline used for a multi-candidate configuration. A single controller is not a separately-maintained code path or a limit case of "up to 3 controllers" — it is the candidates_count=1 case of the one HA-candidate mechanism, and observable behavior (log lines, error handling, retry/reconnect semantics) shall be identical in shape to the multi-candidate case, just with one candidate.
6. If every configured candidate is unreachable or rejects the join at startup, then the component shall report a clear startup failure identifying which candidates were attempted.
7. The maximum number of controller candidates a component can accept shall be defined by a single project-wide constant (MAX_HA_CANDIDATES in src/include/config.h), not independently duplicated or re-defined elsewhere. N_MAX_MASTERS (src/include/client.h) shall derive its value from MAX_HA_CANDIDATES rather than being a separately-maintained constant that can silently diverge from it, as it does today (3 vs. 4).

Requirement 2: Automatic reconnect after controller failover

Objective: As an HA cluster operator, I want scheduler, worker, and apiserver to automatically reconnect to a surviving controller candidate when their active controller connection is lost, so that a controller failover does not require manually restarting these daemons.

Acceptance Criteria

1. When a component's active connection to its controller is lost, the component shall attempt to reconnect using the complete configured candidate list, not only the previously connected candidate.
2. While reconnecting, the component shall keep retrying across the full configured candidate list until one candidate accepts the connection as primary or the component is shut down by an operator.
3. If a reconnect sweep exhausts all configured candidates without success, then the component shall continue retrying rather than permanently giving up, so the component self-heals once a primary becomes reachable again.
4. When apiserver reconnects to a new primary after a controller failover, apiserver shall resume receiving backend push notifications without requiring an apiserver process restart.
5. If the application-level re-join call issued after a successful transport-level reconnect itself fails (including its own retry-budget exhaustion), the component shall retry that re-join rather than treat the failure as final; a pending-rejoin indicator shall be cleared only once the re-join call succeeds, not merely once it has been attempted, so a component is never left transport-connected but silently un-joined to the cluster.

Requirement 3: Reconnect retry does not block other liveness-critical work

Objective: As an HA cluster operator, I want the worker and scheduler's controller-reconnect retries to never block their other liveness-critical processing, so that a slow or unreachable controller candidate does not make the daemon itself appear unresponsive.

Acceptance Criteria

1. While a worker is retrying a controller connection (including any backoff delay between attempts), the worker shall continue performing its other liveness-critical duties (hardware/process monitoring and job-state tracking) without being blocked by the retry attempt or backoff delay.
2. While a scheduler is retrying a controller connection (including any backoff delay between attempts), the scheduler's allocation/decision processing shall continue briefly past the point the retry begins, then block until connectivity is restored, at which point processing shall resume automatically from exactly where it left off. This guarantee is bounded, not indefinite: the scheduler's picker and allocator threads are buffered against a blocked retry only by the depth of its internal jbuf/decisions queues — a scheduler-core determinism invariant (jbuf capacity fixed at 1) that this feature does not adjust — so allocation/decision processing does stall for the remainder of an outage once that buffer is exhausted. The exact number of jobs this buffering absorbs before stalling depends on buffer occupancy at the moment the outage begins (as few as 1-2, as many as ndecisions+1=6 under today's buffer sizes) — an implementation detail of scheduler-core, not a fixed guarantee this requirement makes. What this requirement does guarantee is that no job picked or decided during that bounded window is ever lost, whether it is still buffered when the outage begins or blocks mid-publish — see Requirement 7.
3. The worker and scheduler's reconnect retries shall not delay or prevent the component's own heartbeat responses to a connected controller, so a stalled reconnect attempt elsewhere cannot itself trigger an unnecessary failover.

Requirement 4: apiserver behavior toward API clients during backend disconnection

Objective: As an API client of apiserver, I want a clear, immediate response when the backend controller connection is down, so that I am not left waiting indefinitely for requests that cannot currently be served.

Acceptance Criteria

1. When apiserver's connection to the controller is down and a reconnect is in progress, apiserver shall respond to API requests that require the backend with a clear service-unavailable style error rather than hanging until a connection is restored.
2. While the backend reconnect is in progress, apiserver shall continue attempting to serve API requests that do not require the backend connection.

Requirement 5: Operator-visible connect/reconnect diagnostics

Objective: As an HA cluster operator, I want visibility into which controller candidates each component tried and why each attempt failed, so that I can distinguish a real outage from a configuration mistake.

Acceptance Criteria

1. When a component attempts a connection or reconnection to a controller candidate, the component shall log the candidate attempted and the outcome (success, unreachable, or standby-redirect).
2. If a component exhausts all configured candidates during a connection or reconnection attempt, then the component shall log a summary identifying every candidate attempted and the reason each failed.

Requirement 6: Consistent candidate-list configuration across components

Objective: As an HA cluster operator, I want to configure the controller candidate list the same way for scheduler, worker, and apiserver, so that I do not need to learn or maintain three different configuration formats for the same setting.

Acceptance Criteria

1. The candidate-list configuration accepted by apiserver shall use the same [name=]host:port[,...] syntax already accepted by scheduler and worker.
2. If a component's candidate-list configuration is malformed, then the component shall reject it with an error identifying the malformed entry, consistent with how the other components report malformed candidate-list configuration.

Requirement 7: Scheduler decision delivery survives extended outages without loss

Objective: As an HA cluster operator, I want the scheduler to never silently lose a job dispatch or pend decision because the controller was unreachable longer than a single retry budget, so that scheduler-local job state and controller-observed job state cannot permanently diverge after a failover.

Background

xcecli_jobdispatch/xcecli_set_pend_reason are wrapped in XCECLI_FAILOVER_CALL, which retries a single call up to a bounded budget (max_retries, default 20; backoff capped at max_retry_ms, default 5000ms — roughly 1-2 minutes total) before returning -E_XCE_NOT_READY (client_failover.c:281-282,312-317). This is a per-call budget, not indefinite retry. Today, the scheduler's decision-drain loop (driver.c's decision-drain loop calling xce_sched_driver_on_decision) does not inspect this return code — it logs and moves on to the next queued decision regardless of outcome. Because sched_commit_decision already mutates scheduler-local job state (job->allocs, accounts) before the publish attempt (scheduler.c sched_allocator), a budget-exhausted publish leaves the scheduler believing a job is dispatched/pending while the controller was never told — a silent, permanent state divergence, not a delay.

Acceptance Criteria

1. When publishing a decision (xcecli_jobdispatch or xcecli_set_pend_reason) returns -E_XCE_NOT_READY (single-call retry budget exhausted) while the scheduler has not been shut down, the scheduler shall retry publishing that same decision rather than discarding it or advancing to the next queued decision.
2. The scheduler shall keep retrying a given decision's publish across repeated retry-budget exhaustion until either the publish succeeds or the scheduler is shut down by an operator, consistent with Requirement 2 AC3's "never permanently give up" guarantee applied at the level of an individual decision, not only the connection.
3. Decisions already queued behind a stuck decision shall not be published out of order — the scheduler's existing decision-ordering guarantee is preserved while a decision is being retried.
4. When a decision publish enters extended retry (one or more budget exhaustions) and when it eventually succeeds, the scheduler shall log this distinctly from an ordinary single-attempt success, so operators can distinguish "delayed but delivered" from "delivered on the first try."
5. This requirement applies to the scheduler's own decision-publishing direction only; it does not extend to worker job-state resync on rejoin (owned by the separate worker-rejoin-resync backlog item) or to any controller-side redelivery mechanism.
6. This requirement guarantees delivery only for as long as the scheduler process itself stays running through the outage — decisions held in the in-memory decisions/jbuf queues (including the one actively being retried) do not survive a scheduler process crash or restart, since no persistence mechanism for queued-but-unpublished decisions exists today or is added by this requirement. A scheduler restart during an outage loses the same in-flight decisions it would have lost before this requirement, and requires a fresh scheduling pass after reconnecting — this requirement closes the silent-drop-while-still-running gap, not a durability gap.