save status

.kiro/specs/component-failover-adoption/design.md

@@ -0,0 +1,505 @@
+# Design Document: component-failover-adoption
+
+## Overview
+
+Scheduler, worker, and apiserver each connect to the controller cluster today using only a single configured endpoint at a time, with divergent, partly-broken hand-rolled logic for trying alternates. Meanwhile `libxce` already ships a complete, tested, transparent multi-master mechanism (`XCECLI_FAILOVER_CALL` and friends in `src/lib/client.c`) — the same one `xce_test`'s `masters` setting already uses correctly. This design wires all three components onto that existing mechanism: populate the full candidate list once, let the library rotate/retry/reconnect on every call, and register a rejoin callback so a component re-establishes cluster membership after a transparent reconnect.
+
+**Users**: HA cluster operators who configure multiple controller candidates (`--peers` for scheduler/worker, `backend.peers` YAML for apiserver) and expect scheduler, worker, and apiserver to survive a controller failover without manual restarts — the same expectation `xce_test` already meets.
+
+**Impact**: Removes worker's and scheduler's hand-rolled per-candidate join loops (deleting, not adding, most of the connection-retry code in `controller_connector.c` and `scheduler.c`); replaces apiserver's Go-level per-peer loop with a single library-driven connect; adds one new small shared C helper and a small Go-side health guard. No changes to the controller or to `libxce`'s failover mechanism itself.
+
+### Goals
+- Scheduler, worker, and apiserver each hand the full configured candidate list to `xce_client_config_t` on the very first connection attempt (Req 1).
+- All three automatically reconnect and resume cluster membership after their controller connection is lost, without a process restart (Req 2).
+- Worker's and scheduler's retry/backoff never blocks duties independent of controller connectivity (Req 3).
+- apiserver returns a clear, fast error to its own API clients while its backend connection is down, instead of hanging (Req 4).
+- Every connect/reconnect attempt and outcome is logged per-candidate (Req 5).
+- apiserver accepts the identical candidate-list syntax scheduler/worker already accept (Req 6).
+
+### Non-Goals
+- Changing `xce_test` (it is already the reference-correct client).
+- Completing or adopting the separate, unused `xce_connect_cluster()`/`xce_reconnect_*` primitive in `cluster.c` — superseded by the mechanism this design adopts; left untouched.
+- Raising the `N_MAX_MASTERS` (3) cap or reconciling it with `MAX_HA_CANDIDATES` (4) — pre-existing constraint, replicated (clamp + WARN) exactly as `xce_test` already does.
+- Wiring CA-cert/TLS trust into the real connect path — `xce_client_config_t` carries no such field today and this feature does not add one; TLS is established the same way it already is.
+- Controller-side changes (redirect-reply population, peer authentication) — already shipped by `libxce-failover`.
+- Recovering worker job-state lost during a disconnect window — owned by `worker-rejoin-resync`.
+
+## Boundary Commitments
+
+### This Spec Owns
+- A new shared libxce helper that converts a validated `xce_cluster_config_t` (parsed candidate list) into a fully-populated `xce_client_config_t`.
+- Worker's, scheduler's, and apiserver's integration code that calls `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` with the full candidate list and registers a rejoin callback.
+- apiserver's candidate-list config parsing (routed through the existing C parser via CGO instead of a second Go implementation).
+- apiserver's client-facing fail-fast behavior while its backend connection is degraded.
+- Connect/reconnect diagnostic logging in all three components.
+
+### Out of Boundary
+- `xce_test` and its `masters` setting (unchanged, reference implementation).
+- `libxce`'s failover mechanism itself (`client.c`, `client_internal.h`, `client_failover.c`) — consumed as-is, not modified.
+- `xce_connect_cluster()`/`xce_reconnect_*` (unused, untouched).
+- Controller-side membership/redirect logic.
+- Worker job-state resync content on rejoin (`worker-rejoin-resync`).
+- `N_MAX_MASTERS`/`MAX_HA_CANDIDATES` value changes.
+- TLS/CA-cert plumbing into the real connect path.
+
+### Allowed Dependencies
+- `src/lib/client.c` / `client_internal.h` / `client_failover.h` (shipped `libxce-failover` mechanism) — read-only dependency, no modification.
+- `src/lib/cluster.c` / `xce/cluster.h` (`xce_parse_peers`, `xce_cluster_config_validate`) — existing, unmodified; this spec adds one new function alongside them.
+- Existing per-component candidate-list CLI/YAML surfaces (`--peers`, `--ca-cert`, `backend.peers`) — unmodified externally; only how they're consumed changes.
+
+### Revalidation Triggers
+- Any change to `xce_client_config_t`'s layout or `N_MAX_MASTERS`.
+- Any change to `XCECLI_FAILOVER_CALL`/`XCECLI_FF_CALL` retry/classification semantics (`libxce-failover`).
+- Any change to `xce_parse_peers`/`xce_cluster_config_validate` syntax or validation rules (apiserver's CGO-based parser would need re-verification).
+- `worker-rejoin-resync` landing — worker's rejoin callback becomes the natural place to also trigger job-state resync; this spec's callback is a no-op-beyond-rejoin placeholder for that.
+
+## Architecture
+
+### Existing Architecture Analysis
+Today each of the three components independently drives its own `xcecli_ctx_t` lifecycle with `nmasters` hardcoded to `1` per connection attempt, and layers its own outer candidate loop on top:
+- **worker**: `controller_connector.c`'s `xcecd_join_cluster` loops candidates itself, but the very first connection (inside `xcecd_start()`) happens before `cc_ctx->cluster_cfg` is even populated (`main.c` assigns it after `xcecd_start()` returns) — the candidate list is silently ignored on startup.
+- **scheduler**: `scheduler.c`'s `sched_init` loops candidates correctly at startup, but there is no reconnect logic anywhere afterward.
+- **apiserver**: `clib.go`'s `RegisterToBackend` loops candidates once via a hand-written Go parser, only at process startup.
+
+None of the three ever set `nmasters > 1` on a single `xce_client_config_t`, so the library's own rotation/retry (`client.c`, wrapped by `XCECLI_FAILOVER_CALL`) never actually engages for them.
+
+### Architecture Pattern & Boundary Map
+
+```mermaid
+graph TB
+    subgraph Daemons
+        Worker[worker xcecd]
+        Scheduler[xce_scheduler]
+        Apiserver[apiserver backend package]
+    end
+
+    subgraph NewSharedCode
+        Convert[xce_cluster_config_to_client_config]
+    end
+
+    subgraph ExistingLibxce
+        Parse[xce_parse_peers]
+        Validate[xce_cluster_config_validate]
+        CtxInit[xcecli_ctx_init xcecli_start xcecli_join]
+        Failover[XCECLI_FAILOVER_CALL rotate retry backoff]
+        Rejoin[rejoin callback xcecli_set_reconnect_handler]
+    end
+
+    subgraph Cluster
+        Primary[Controller Primary]
+        Standby1[Controller Standby]
+    end
+
+    Worker --> Parse
+    Scheduler --> Parse
+    Apiserver --> Parse
+    Parse --> Validate
+    Validate --> Convert
+    Convert --> CtxInit
+    CtxInit --> Failover
+    Failover --> Rejoin
+    Failover --> Primary
+    Failover --> Standby1
+```
+
+**Architecture Integration**:
+- Selected pattern: thin per-component integration layer over an unchanged, already-shipped library mechanism (adapter, not new subsystem).
+- Domain boundaries: `libxce` owns candidate parsing, validation, connect/rotate/retry/backoff, and rejoin-hook invocation (unchanged). Each daemon owns only: building the candidate list from its own config surface, calling the library once, and supplying its role-specific rejoin behavior (what to send when re-joining).
+- Existing patterns preserved: the `xce_parse_peers`/`xce_cluster_config_validate`/`xce_client_config_t` split; the CGO bridge pattern already used throughout `backend/`; worker's dedicated connector thread.
+- New components: one C helper (candidate-list → client-config conversion, generalized out of `xce_test`'s inline logic) and one small Go health-state module (apiserver-only, has no C-side equivalent need).
+- Steering compliance: no new blocking work introduced on iomgr/delaymgr-equivalent threads; worker's existing thread isolation is preserved, not altered.
+
+### Rejoin callback constraint: avoiding self-deadlock on `ac->failover.lock`
+
+Verified directly against `client_failover.c:341-411`: `XCECLI_FAILOVER_CALL` holds `ac->failover.lock` (a plain, non-recursive `pthread_mutex_t` — `mylock_init` calls `pthread_mutex_init(l, NULL)`, no `PTHREAD_MUTEX_RECURSIVE` attribute) for the **entire** retry loop, and `xcecli_invoke_rejoin()` — which calls the registered rejoin callback — is invoked **synchronously from inside that loop, while the lock is held** (`client_failover.c:397`). Any rejoin callback that itself calls a public `xcecli_*` wrapper on the *same* `xcecli_ctx_t` (e.g. `xcecli_join`, which is itself `XCECLI_FAILOVER_CALL`-wrapped and tries to re-lock the same mutex) **self-deadlocks** — the calling thread blocks forever on its own already-held lock. This was never caught because `xcecli_set_reconnect_handler` has zero production callers today and the existing unit test's callback is a no-op probe that never calls back into the client.
+
+This directly constrains every component's rejoin implementation in this spec:
+- The callback **registered with the library** (worker's `checking_connection`→`xcecli_hello` path, scheduler's `xcecli_jobdispatch` path, apiserver's write-call path) must do **no I/O and call no `xcecli_*` function** — it may only set a lightweight "needs rejoin" flag (a plain bool/atomic on the consumer side) and return `0` so the interrupted operation resumes normally.
+- Each component's own calling code, **after** the triggering `xcecli_*` call returns (at which point the library has released the lock), checks and clears that flag and issues `xcecli_join(...)` **as a separate, subsequent call** — safe, since the lock is free by then.
+- apiserver's heartbeat-triggered path (see below) is unaffected by this constraint — it already detects recovery and calls the rejoin logic **after** `xcecli_heartbeat` has returned (`XCECLI_FF_CALL` also releases its lock before returning), so it was never at risk.
+
+### Technology Stack
+
+| Layer | Choice / Version | Role in Feature | Notes |
+|-------|------------------|------------------|-------|
+| Shared library | C (gnu17), `src/lib/cluster.c` | New candidate→client-config conversion helper | Extends existing file alongside `xce_parse_peers` |
+| Worker / Scheduler | C (gnu17) | Consume the existing `xcecli_ctx_t` mechanism directly | No new dependency |
+| apiserver | Go 1.23.4 + CGO | Consume the same mechanism via the existing CGO bridge; new Go-side health-state guard | `sync/atomic`, consistent with existing `heartbeat.go` pattern |
+
+## File Structure Plan
+
+### Modified / New Files
+```
+src/include/xce/cluster.h                          # + declare xce_cluster_config_to_client_config()
+src/lib/cluster.c                                    # + implement xce_cluster_config_to_client_config()
+src/lib/testsuite/test_cluster.c                      # + unit tests for the new helper
+
+src/worker/src/controller_connector.c                # rewrite: delete per-candidate loop in
+                                                       #   xcecd_join_cluster/init_client_config;
+                                                       #   single init+start+join using full
+                                                       #   candidate list; register rejoin callback
+src/worker/src/main.c                                 # fix: populate cc_ctx->cluster_cfg BEFORE
+                                                       #   xcecd_start() (removes init-order bug)
+src/worker/include/controller_connector.h             # + rejoin callback function declaration
+
+src/scheduler/src/scheduler.c                         # rewrite: sched_init uses the new helper +
+                                                       #   single init+start+join; delete inline
+                                                       #   per-candidate loop; register rejoin callback
+src/scheduler/src/main.c                              # no ordering change needed (already correct);
+                                                       #   simplified call site only
+
+src/frontend/apiserver/backend/clib.go                # rewrite RegisterToBackend: single connect
+                                                       #   using the full candidate list via CGO calls
+                                                       #   into xce_parse_peers/xce_cluster_config_validate/
+                                                       #   xce_cluster_config_to_client_config; cgo
+                                                       #   preamble gains a static C shim (mirrors the
+                                                       #   existing xce_backend_handler/go_push_handler
+                                                       #   pattern) that calls the reconnect.go export,
+                                                       #   passed to xcecli_set_reconnect_handler; the
+                                                       #   heartbeat tick callback explicitly re-invokes
+                                                       #   the same rejoin path on recovery (see below)
+src/frontend/apiserver/backend/health.go              # NEW: backend-degraded circuit-breaker state
+src/frontend/apiserver/backend/reconnect.go           # NEW: exported Go rejoin logic (re-sends
+                                                       #   xcecli_join for HostRole_apiserver, clears
+                                                       #   health.go state); called from clib.go's C
+                                                       #   shim AND directly from the heartbeat callback
+src/frontend/apiserver/backend/heartbeat.go           # MODIFIED: tick callback detects a
+                                                       #   degraded-to-healthy transition after
+                                                       #   xcecli_heartbeat succeeds and explicitly
+                                                       #   triggers reconnect.go's rejoin logic
+src/frontend/apiserver/utils/config.go                # no schema change (Peers already exists)
+
+src/worker/testsuite/test_e2e_multimaster_join.c       # NEW: candidate-list join + reconnect E2E
+src/scheduler/testsuite/test_e2e_multimaster_join.c    # NEW: same, scheduler
+src/frontend/apiserver/backend/health_test.go          # NEW: circuit-breaker unit tests
+src/frontend/apiserver/backend/clib_test.go            # + candidate-list parity tests (existing file)
+```
+
+> `controller/` and `libxce`'s failover mechanism itself are not modified — only consumed.
+
+## System Flows
+
+### Startup connect (Req 1)
+
+```mermaid
+sequenceDiagram
+    participant Daemon
+    participant Convert as xce_cluster_config_to_client_config
+    participant Ctx as xcecli_ctx (client.c)
+    participant CandA as Candidate A (standby)
+    participant CandB as Candidate B (primary)
+
+    Daemon->>Convert: full candidate list (validated)
+    Convert-->>Daemon: xce_client_config_t (nmasters=N)
+    Daemon->>Ctx: xcecli_ctx_init + xcecli_start
+    Ctx->>CandA: connect
+    CandA-->>Ctx: connected, not primary
+    Ctx->>Daemon: xcecli_join (role)
+    Ctx->>CandA: join request
+    CandA-->>Ctx: standby redirect
+    Ctx->>Ctx: rotate to next candidate
+    Ctx->>CandB: connect + join
+    CandB-->>Ctx: joined as primary
+    Ctx-->>Daemon: xcecli_join success
+```
+
+### Runtime failover (Req 2, Req 3)
+
+```mermaid
+sequenceDiagram
+    participant Daemon
+    participant Ctx as xcecli_ctx lock held
+    participant OldPrimary as Old Primary dies
+    participant NewPrimary as New Primary
+
+    Daemon->>Ctx: xcecli_* call existing traffic
+    Ctx->>OldPrimary: send
+    OldPrimary--xCtx: connection lost
+    Ctx->>Ctx: XCECLI_FAILOVER_CALL rotates and backs off
+    Ctx->>NewPrimary: reconnect
+    NewPrimary-->>Ctx: connected
+    Ctx->>Daemon: invoke rejoin callback lock still held
+    Note over Daemon: callback only sets a needs-rejoin flag, no I/O, returns 0
+    Ctx->>NewPrimary: resume original call
+    NewPrimary-->>Ctx: reply
+    Ctx-->>Daemon: original call result, lock released
+    Note over Daemon: after the call returns, lock is free
+    Daemon->>Daemon: check and clear needs-rejoin flag
+    Daemon->>Ctx: xcecli_join role separate call
+    Ctx->>NewPrimary: join
+    NewPrimary-->>Ctx: joined
+    Ctx-->>Daemon: rejoin complete
+```
+
+Key decision reflected in the diagram: the rejoin callback invoked **by the library** never performs I/O — it only flags that a rejoin is needed (see "Rejoin callback constraint" above, avoiding a self-deadlock on the still-held per-ctx lock). The actual `xcecli_join` call happens as a distinct follow-up call from the daemon, on the same thread (worker: connector thread; scheduler: allocator thread; apiserver: the requesting goroutine), once the triggering call has returned and released the lock — no separate reconnect loop or new thread is introduced either way.
+
+### apiserver fail-fast while degraded (Req 4)
+
+```mermaid
+sequenceDiagram
+    participant Client as API Client
+    participant Handler as apiserver handler
+    participant Health as health.go
+    participant Ctx as xcecli_ctx (gCtx)
+
+    Client->>Handler: HTTP request needing backend
+    Handler->>Health: is backend degraded?
+    alt degraded
+        Health-->>Handler: yes
+        Handler-->>Client: 503 service unavailable (fast)
+    else healthy
+        Health-->>Handler: no
+        Handler->>Ctx: backend call
+        alt call exhausts retry budget
+            Ctx-->>Handler: error
+            Handler->>Health: mark degraded (with TTL)
+            Handler-->>Client: 503 service unavailable
+        else call succeeds
+            Ctx-->>Handler: result
+            Handler-->>Client: 200 with result
+        end
+    end
+```
+
+### apiserver idle-period reconnect (Req 2.4)
+
+```mermaid
+sequenceDiagram
+    participant HB as heartbeat.go tick
+    participant Ctx as xcecli_ctx (gCtx, FF_CALL)
+    participant Rejoin as reconnect.go
+    participant Health as health.go
+    participant NewPrimary as New Primary
+
+    Note over HB: no client requests in flight
+    HB->>Ctx: xcecli_heartbeat (fire-and-forget)
+    Ctx->>Ctx: connection known-dead: best-effort rotate + reconnect
+    Ctx->>NewPrimary: reconnected transport only
+    Ctx-->>HB: rc success (was failing, now succeeds)
+    HB->>HB: detect failure-to-success transition
+    HB->>Rejoin: explicitly trigger rejoin
+    Rejoin->>NewPrimary: xcecli_join(HostRole_apiserver)
+    NewPrimary-->>Rejoin: joined
+    Rejoin->>Health: MarkHealthy()
+```
+
+This path exists because `XCECLI_FF_CALL` (fire-and-forget, used by the heartbeat) never invokes the library's own rejoin hook — only the full-retry path (`XCECLI_FAILOVER_CALL`, used by write calls) does. Without this explicit trigger, an idle apiserver instance would repair its TCP connection on every heartbeat tick but never re-establish cluster membership, so push notifications would stay dead until an unrelated write request happened to arrive.
+
+## Requirements Traceability
+
+| Requirement | Summary | Components | Interfaces | Flows |
+|-------------|---------|------------|------------|-------|
+| 1.1-1.3 | Try each candidate in order at startup | libxce (unchanged), ConfigConverter | `xcecli_start`, `xcecli_join` | Startup connect |
+| 1.4 | Worker uses full list on FIRST attempt | WorkerControllerConnector | `controller_connector.c` init order fix | Startup connect |
+| 1.5 | Single/no-candidate mode unchanged | All three | ConfigConverter (`nmasters=1` path) | — |
+| 1.6 | Clear startup failure if all candidates fail | All three | existing `xcecli_start`/`xcecli_join` return codes | Startup connect |
+| 2.1-2.3 | Automatic reconnect + retry until success or shutdown | libxce (unchanged), all three's rejoin callbacks | `xcecli_set_reconnect_handler` | Runtime failover |
+| 2.4 | apiserver resumes push notifications after reconnect (incl. idle-period recovery) | ApiserverBackendBridge | `reconnect.go`'s actual-rejoin function re-sends `xcecli_join(HostRole_apiserver)`, triggered via the shared post-call helper's flag check on write-path reconnect AND explicitly from `heartbeat.go` on idle-period recovery | Runtime failover, apiserver idle-period reconnect |
+| 3.1-3.2 | Retry/backoff doesn't block other liveness work | WorkerControllerConnector (thread isolation preserved), SchedulerControllerClient (allocator/picker thread separation) | existing thread structure | Runtime failover |
+| 4.1-4.2 | apiserver fails fast to clients while degraded | ApiserverBackendHealth | `health.go` | apiserver fail-fast flow |
+| 5.1-5.2 | Per-candidate connect/reconnect logging | All three | log statements at each rotation | Startup connect, Runtime failover |
+| 6.1-6.2 | Consistent candidate-list syntax | ApiserverConfigParity | CGO call to `xce_parse_peers`/`xce_cluster_config_validate` | Startup connect |
+
+## Components and Interfaces
+
+| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies | Contracts |
+|-----------|--------------|--------|---------------|-------------------|-----------|
+| ConfigConverter | libxce (C) | Convert validated candidate list to client config | 1.1-1.6, 6.1 | `xce_cluster_config_validate` (P0) | Service |
+| WorkerControllerConnector | worker (C) | Single init/connect/join + rejoin for worker | 1.1-1.4, 2.1-2.3, 3.1-3.2, 5.1-5.2 | ConfigConverter (P0), `xcecli_*` (P0) | Service |
+| SchedulerControllerClient | scheduler (C) | Single init/connect/join + rejoin for scheduler | 1.1-1.3, 1.5-1.6, 2.1-2.3, 3.1-3.2, 5.1-5.2 | ConfigConverter (P0), `xcecli_*` (P0) | Service |
+| ApiserverBackendBridge | apiserver (Go/CGO) | Single init/connect/join for apiserver; config parity | 1.1-1.6, 2.4, 5.1-5.2, 6.1-6.2 | ConfigConverter via CGO (P0), `xcecli_*` via CGO (P0) | Service |
+| ApiserverBackendHealth | apiserver (Go) | Fail-fast circuit breaker for backend-down state | 4.1-4.2 | ApiserverBackendBridge (P0, consumes rejoin/error signals) | State |
+
+### Shared Library
+
+#### ConfigConverter
+
+| Field | Detail |
+|-------|--------|
+| Intent | Turn a validated `xce_cluster_config_t` into a fully-populated `xce_client_config_t`, clamped to `N_MAX_MASTERS` with a WARN on truncation |
+| Requirements | 1.1, 1.2, 1.3, 1.5, 1.6, 6.1 |
+
+**Responsibilities & Constraints**
+- Pure, socket-free transformation (mirrors `xce_test`'s existing inline logic at `test.c:360-372`, generalized into one reusable function).
+- Does not call `xce_cluster_config_validate()` itself — callers validate first (existing contract, unchanged) and pass a pre-validated config.
+- Does not own `from`/`key` (identity/auth) — caller sets those on the output struct after conversion, matching how `xce_test`, worker, and scheduler already source them independently (CLI/env-specific).
+
+**Dependencies**
+- Inbound: WorkerControllerConnector, SchedulerControllerClient, ApiserverBackendBridge (all P0)
+- Outbound: none (leaf function)
+
+**Contracts**: Service [x]
+
+##### Service Interface
+```c
+/* xce/cluster.h */
+/**
+ * xce_cluster_config_to_client_config() — convert a validated candidate list
+ * into an xce_client_config_t's master fields (nmasters/masters/master_ports).
+ *
+ * Copies up to N_MAX_MASTERS candidates from cfg->candidates[] in order.
+ * If cfg->candidates_count > N_MAX_MASTERS, copies the first N_MAX_MASTERS and
+ * logs a WARN identifying the truncated count (mirrors xce_test's existing
+ * clamp behavior).
+ *
+ * Does NOT set out->from / out->key — caller sets those separately.
+ * Does NOT validate cfg — caller must have already called
+ * xce_cluster_config_validate() successfully.
+ *
+ * Returns 0 on success, -EINVAL if cfg or out is NULL or cfg->candidates_count < 1.
+ */
+int xce_cluster_config_to_client_config(const xce_cluster_config_t* cfg,
+                                        xce_client_config_t* out);
+```
+- Preconditions: `cfg` already passed `xce_cluster_config_validate()`.
+- Postconditions: `out->nmasters` in `[1, N_MAX_MASTERS]`; `out->masters[i]`/`out->master_ports[i]` populated for `i < out->nmasters`, in the same order as `cfg->candidates[]`.
+- Invariants: never writes beyond `N_MAX_MASTERS` entries regardless of `cfg->candidates_count`.
+
+### Worker
+
+#### WorkerControllerConnector
+
+| Field | Detail |
+|-------|--------|
+| Intent | Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect |
+| Requirements | 1.1-1.4, 2.1-2.3, 3.1-3.2, 5.1-5.2 |
+
+**Responsibilities & Constraints**
+- Replaces `xcecd_join_cluster`'s per-candidate loop (`controller_connector.c:237-311`) with: build `xce_client_config_t` via ConfigConverter once (candidates from `cc_ctx->cluster_cfg` when `--peers` given, else today's single-host fallback path unchanged per Req 1.5), then one `xcecli_ctx_init` + `xcecli_start` + `xcecli_join`.
+- Fixes the init-order bug by moving `cc_ctx->cluster_cfg` population (`main.c`) to before `xcecd_start()` is called, so the very first connect already sees the full candidate list (Req 1.4).
+- Registers a rejoin callback (`xcecli_set_reconnect_handler`) that only sets a `needs_rejoin` flag and returns `0` — per "Rejoin callback constraint" above, it must not call `xcecli_join` (or any `xcecli_*` function) directly, since the library invokes it while `ac->failover.lock` is still held.
+- `checking_connection()`'s periodic `xcecli_hello` keeps its existing role (liveness probe on the connector thread) but no longer needs to manually call `xcecd_join_cluster` on failure — a transport failure inside `xcecli_hello` already triggers the library's transparent reconnect + the registered flag-setting callback. Immediately after each `xcecli_hello` call returns (lock now free), `checking_connection()` checks and clears `needs_rejoin`; if set, it calls `xcecli_join(ctx, HostRole_worker, &host_attrs)` as a separate call — the same join payload used at startup.
+- Runs entirely on the existing dedicated connector thread (`start_controller_connector_thread`/`connector_to_controller`), preserving isolation from `hardware_monitor`/`system_monitor`/`process_manager` update paths (Req 3.1) — no new thread introduced.
+
+**Dependencies**
+- Inbound: `xcecd_start()` (P0)
+- Outbound: ConfigConverter (P0), `xcecli_*` (P0)
+
+**Contracts**: Service [x]
+
+**Implementation Notes**
+- Integration: preserve existing WARN-level log statements around standby-redirect and candidate-exhaustion, now emitted from the library's classification path or immediately around the single init/join call, so Req 5 diagnostics are not lost in the simplification.
+- Validation: implementation must confirm (code read, not test-only) that `hardware_monitor_update`/`system_monitor_update`/`process_manager_update` do not execute on the connector thread — flagged as a Validation Hook, not re-verified in this design pass.
+- Risks: deleting the hand-rolled loop must not silently drop the `MAX_RETRY_COUNT`/`RETRY_INTERVAL` bounded-retry-then-`handle_disconnect()` behavior in `checking_connection()` — that outer bound (Req 2.3's "keep retrying" vs. any existing bounded-give-up behavior) must be reconciled explicitly during implementation, not left ambiguous.
+
+### Scheduler
+
+#### SchedulerControllerClient
+
+| Field | Detail |
+|-------|--------|
+| Intent | Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect |
+| Requirements | 1.1-1.3, 1.5-1.6, 2.1-2.3, 3.1-3.2, 5.1-5.2 |
+
+**Responsibilities & Constraints**
+- Replaces `sched_init`'s inline per-candidate loop (`scheduler.c:462-495`) with ConfigConverter + one `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` call.
+- Registers a rejoin callback that only sets a `needs_rejoin` flag and returns `0` (same constraint as worker — the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join` itself). After each `xcecli_jobdispatch` call returns, the allocator checks and clears the flag; if set, it calls `xcecli_join(ctx, HostRole_scheduler, NULL)` as a separate call before continuing.
+- No new thread: `xcecli_jobdispatch` (the only backend call site, `scheduler.c:1289`) continues to run from the existing `sched_allocator` thread; blocking during a reconnect/backoff cycle is an accepted, benign degradation since the allocator has no useful work without a controller anyway, and `sched_job_picker` runs on its own separate thread unaffected (Req 3.1-3.2 satisfied by the existing thread split, not new code).
+
+**Dependencies**
+- Inbound: `sched_init()` (P0)
+- Outbound: ConfigConverter (P0), `xcecli_*` (P0)
+
+**Contracts**: Service [x]
+
+**Implementation Notes**
+- Integration: preserve the existing `WARN`/`INFO` log lines around candidate parsing (`main.c`) and add per-attempt/per-outcome logging around the single init/join call (today's loop logged each attempt; the library's own classification path should be checked for equivalent visibility, else add it locally).
+- Risks: today's total absence of reconnect logic means this is additive behavior change (scheduler previously required a manual restart after primary loss) — call out explicitly in the task/rollout notes so operators aren't surprised by scheduler now retrying indefinitely instead of exiting.
+
+### apiserver
+
+#### ApiserverBackendBridge
+
+| Field | Detail |
+|-------|--------|
+| Intent | Single connect/join at startup using the full candidate list via the C library (through CGO), replacing the Go-level per-peer loop; config-syntax parity with scheduler/worker |
+| Requirements | 1.1-1.6, 2.4, 5.1-5.2, 6.1-6.2 |
+
+**Responsibilities & Constraints**
+- `RegisterToBackend()` (`clib.go`) is rewritten: build a C `xce_cluster_config_t` from `config.Peers` (or the single `Address`/`Port` fallback, Req 1.5) by calling `xce_parse_peers` via CGO (not the existing hand-written Go `parsePeer`), validate with `xce_cluster_config_validate`, convert with ConfigConverter, then call `xce_backend_init` **once** with the resulting `nmasters > 1` config — replacing the current `for _, peer := range config.Peers { tryConnect(...) }` Go loop entirely.
+- `reconnect.go` holds two plain Go functions: (1) a `//export`-annotated flag-setter — registered as the library's rejoin callback, does no I/O, only sets a `needs_rejoin` atomic flag and returns `0` (per "Rejoin callback constraint" above: the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join`/any CGO backend function itself); (2) a plain Go function that performs the actual rejoin — calls `xcecli_join(gCtx.ctx, HostRole_apiserver, NULL)` and clears `ApiserverBackendHealth`'s degraded state on success.
+- The flag-setter is **registered with the C library from `clib.go`'s own cgo preamble**, via a small static C shim (`xce_backend_rejoin_handler`, mirroring the existing `xce_backend_handler`/`go_push_handler` pattern already used for push dispatch) passed to `xcecli_set_reconnect_handler` — cgo symbol resolution is per-file (see `wire_dispatch_support.go`'s documented rationale for the identical `go_push_handler` wrapper), so the shim and the registration call must live in the same file as the `extern` declaration, not standalone in `reconnect.go`.
+- Rather than duplicating the same pre-call/post-call logic across all four wrapper functions, the pre-call `IsDegraded()` fail-fast check (ApiserverBackendHealth, below) and the post-call `needs_rejoin` check-and-clear are both implemented once, in a single shared internal helper each of the four `clib.go` wrappers calls around its `C.xcecli_*` invocation — not copy-pasted per wrapper.
+- **Idle-period reconnect gap (Req 2.4) fix**: `XCECLI_FF_CALL` — the macro backing `xcecli_heartbeat`, `heartbeat.go`'s existing periodic prober (`cmd/main.go:76`) — performs a best-effort transport reconnect on a known-dead connection but never invokes the library's rejoin hook (verified against `client_internal.h`/`client_failover.c`: `xcecli_invoke_rejoin` is called only from `xcecli_retry_next`, the full-retry path used by write calls, never from the fire-and-forget path). Left as-is, a controller failover occurring while apiserver has no in-flight write request would silently repair the TCP connection without ever re-joining, permanently starving push notifications until an unrelated write request happens to arrive. Fix: the heartbeat tick callback (`heartbeat.go`) tracks the previous tick's outcome; when a tick transitions from failure to success (transport just recovered), it explicitly calls `reconnect.go`'s rejoin function itself, then clears `ApiserverBackendHealth`. This reuses the same rejoin logic Task 4.3 already builds — no duplicate join-payload code, just an explicit trigger point the write-call path already gets for free from the library.
+
+**Dependencies**
+- Inbound: `cmd/main.go` (P0)
+- Outbound: ConfigConverter via CGO (P0), `xcecli_*` via CGO (P0), ApiserverBackendHealth (P1, signals it on outcome)
+
+**Contracts**: Service [x]
+
+##### Service Interface
+```go
+// backend/clib.go
+// RegisterToBackend initialises the client using the full configured
+// candidate list (config.Peers, or the single Address/Port fallback) and
+// registers the rejoin callback. Single call — no per-peer Go loop.
+func RegisterToBackend(config *BackendConfig) error
+```
+- Preconditions: `StartNotificationListener` already called (unchanged from today).
+- Postconditions: on success, `gCtx.ctx` is joined with `nmasters` reflecting every configured candidate (clamped per ConfigConverter); the rejoin callback is registered.
+- Invariants: candidate-list parsing/validation errors surface the same way `xce_parse_peers`/`xce_cluster_config_validate` report them in C (Req 6.2) — not a separately-worded Go error.
+
+**Implementation Notes**
+- Integration: `parsePeer` (Go) and its tests are removed/retired in favor of the CGO-routed parser (Req 6.1).
+- Integration: the rejoin C shim must be defined in `clib.go`'s cgo preamble (not `reconnect.go`) — see Responsibilities above; this was the design's one build-risk item, now resolved by construction rather than left as an implementation-time surprise.
+- Risks: CGO error surfaces (C `-EINVAL` etc.) need a clear Go-side error message mapping so operators get an actionable message (Req 1.6, 6.2), not a bare error code.
+
+#### ApiserverBackendHealth
+
+| Field | Detail |
+|-------|--------|
+| Intent | Let request handlers fail fast (503) instead of blocking on the library's full retry budget when the backend is known-degraded |
+| Requirements | 4.1, 4.2 |
+
+**Responsibilities & Constraints**
+- New `health.go`: an atomic "degraded-until" timestamp. `IsDegraded() bool` — true if `now < degraded-until`. `MarkDegraded(ttl time.Duration)` — called by backend-call wrapper functions in `clib.go` when a call's returned error indicates retry-budget exhaustion (e.g. `E_XCE_NOT_READY`). `MarkHealthy()` — called by the rejoin callback (`reconnect.go`) on a successful rejoin.
+- Backend-call wrapper functions used by request-handling code paths (job submit, job control, queue control, host control, etc. — the existing `gCtx.ctx`-based calls in `clib.go`) check `IsDegraded()` first, via the shared internal helper described under ApiserverBackendBridge above (not duplicated per wrapper); if true, return a clear "backend unavailable" error immediately instead of invoking the blocking C call (Req 4.1). Calls that don't require the backend are unaffected (Req 4.2 — nothing to change there, they already don't touch `gCtx`).
+
+**Dependencies**
+- Inbound: ApiserverBackendBridge (sets state on outcomes)
+- Outbound: none
+
+**Contracts**: State [x]
+
+##### State Management
+- State model: single atomic degraded-until timestamp (int64 unix-nano via `sync/atomic`, same pattern as existing `heartbeatIntervalSecs` in `heartbeat.go`).
+- Persistence & consistency: process-local, in-memory only; reset on apiserver restart (matches the rest of apiserver's in-memory state model per `tech.md`).
+- Concurrency strategy: lock-free atomic read/write; safe for concurrent goroutines per request.
+
+**Implementation Notes**
+- Integration: TTL-based clearing (Risks & Mitigations in research.md) in addition to the rejoin-callback clear, so a missed/edge-case rejoin doesn't permanently wedge the flag.
+- Validation: unit tests cover degraded→healthy transitions via both the TTL expiry and explicit `MarkHealthy()`.
+
+## Error Handling
+
+### Error Strategy
+No new error taxonomy — reuse the existing controller status codes (`E_XCE_REDIRECT`/`E_XCE_STANDBY`/`E_XCE_NOT_READY`, `errnox.h`) that `client_failover.c` already classifies, and the existing `xce_cluster_config_validate()` `-EINVAL` cases for malformed candidate-list config.
+
+### Error Categories and Responses
+- **Malformed candidate-list config** (Req 1.6, 6.2): `xce_parse_peers`/`xce_cluster_config_validate` failure at startup → each component reports a clear message naming the offending entry and refuses to start (unchanged behavior for worker/scheduler; apiserver gains the same via the CGO route).
+- **All candidates unreachable at startup** (Req 1.6): `xcecli_start`/`xcecli_join` returns the last transport error after exhausting all candidates → component reports which candidates were attempted (Req 5.2) and fails startup (worker/scheduler: process exits as today; apiserver: `RegisterToBackend` returns an error, `cmd/main.go`'s existing handling applies unchanged).
+- **Runtime disconnect** (Req 2): transparently retried/rotated by the library; components only observe it through their own periodic calls' return codes and the rejoin callback — no new error category, just new logging around the existing outcomes.
+- **Candidate count exceeds `N_MAX_MASTERS`**: ConfigConverter clamps and WARNs (not an error) — matches `xce_test`'s existing behavior exactly.
+- **apiserver backend degraded** (Req 4.1): mapped to a clear 503-style API error, distinct from other 5xx causes, so operators/clients can distinguish "backend down, retrying" from an unrelated internal error.
+
+### Monitoring
+Per-candidate connect/reconnect attempts and outcomes are logged (Req 5) using each component's existing logger (`XCECD_LOG` for worker, the scheduler's existing `INFO`/`WARN` macros, `zap` for apiserver) — no new logging infrastructure, just new call sites around the simplified connect/reconnect path.
+
+## Testing Strategy
+
+- **Unit Tests**:
+  - `ConfigConverter`: clamps at `N_MAX_MASTERS` with WARN when given more candidates; preserves candidate order; rejects `NULL`/empty input (`test_cluster.c`).
+  - apiserver `health.go`: degraded→healthy via `MarkHealthy()`, via TTL expiry; concurrent goroutine safety (`health_test.go`).
+  - apiserver CGO-routed parser: same malformed-input cases scheduler/worker already reject, now asserted through the Go call site (`clib_test.go`).
+  - Rejoin flag-setter callbacks (worker, scheduler, apiserver): with a real `ac->failover.lock` held (simulating the library's call context per "Rejoin callback constraint"), invoking the registered callback returns promptly without blocking and performs no I/O — a regression guard against reintroducing the self-deadlock this design pass found.
+- **Integration Tests**:
+  - Worker/scheduler startup against a 3-candidate config where candidate[0] is a live standby — assert the component joins as expected without operator intervention (Req 1.1-1.3), extending each component's `testsuite/`.
+  - Worker/scheduler reconnect: kill the currently-joined primary mid-session, assert the component reconnects and re-joins the new primary without restart (Req 2.1-2.3), reusing the project's existing `/xce-deploy-ha` + `/xce-test-ha` pattern already used for controller HA soak testing — this is also the test that would hang (not fail cleanly) if the self-deadlock were reintroduced, so it must run with a wall-clock timeout that fails loud rather than hanging CI.
+- **E2E Tests**:
+  - Full 3-node HA cluster with worker + scheduler + apiserver all configured with the full candidate list; kill the primary; assert all three reconnect, worker resumes job execution reporting, scheduler resumes dispatch, apiserver resumes serving requests and receiving push notifications (Req 2.4) — new `test_e2e_multimaster_join.c` per C component plus an apiserver-side check in the existing HA soak harness.
+  - apiserver idle-period reconnect: kill the primary while apiserver has no in-flight write request (only the heartbeat prober running); assert apiserver still rejoins and resumes push notifications, proving the heartbeat-triggered rejoin path (not just the write-call-triggered one) works (Req 2.4).
+  - apiserver fail-fast: with the backend killed and not yet reconnected, assert an API request returns quickly (bounded, sub-second) with a clear 503 rather than hanging for the full retry budget (Req 4.1).
+
+## Security Considerations
+No new trust boundary: candidate hosts are the same set of controllers already trusted today; TLS/identity handling is unchanged (see Non-Goals — CA-cert wiring is out of scope and pre-existing). No new authentication surface is introduced.

.kiro/specs/component-failover-adoption/requirements.md

@@ -0,0 +1,90 @@
+# Requirements Document
+
+## Project Description (Input)
+`component-failover-adoption` — pre-scoped roadmap backlog item (`roadmap.md:64`, recorded 2026-06-16, HARD-dependent on `libxce-failover`, which shipped GO 2026-07-18 — this item is now unblocked). Make the C daemons (scheduler, worker) and the Go apiserver actually exercise libxce's transparent multi-master failover, matching xce_test's `masters`-setting behavior. Two roadmap-defined integration items:
+1. Each component must hand libxce the **full master/candidate list**, not a single endpoint (today even the CLI uses only `conf->masters[0]`).
+2. Bounded retry makes the already-blocking `xce_request` block **longer** (seconds of backoff) — for the async C daemons this must be verified **liveness-safe**: it must never stall a thread that also drives heartbeats (the project-wide "never block iomgr/delaymgr" thread-role contract, `ha-async-io.md`).
+
+Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for **scheduler, worker, apiserver**. `xce_test` is explicitly OUT of scope — it is the already-correct in-spec verification client, not a target.
+
+### Background (verification findings, 2026-07-19)
+
+`xce_test`'s `masters` setting (`src/controller/testsuite/test.c:345-377`) is the reference implementation: it parses a `host:port,...` list via `xce_parse_peers()`, sets `config.nmasters = candidates_count`, fills the full `config.masters[]` array, and calls `xcecli_ctx_init()` once — so libxce's own `client_failover.c` EndpointRotator handles mid-session failover across all candidates.
+
+None of the three production daemons match this pattern today; each has a different gap:
+
+- **worker (xcecd)**: `--peers` is parsed into `cluster_cfg`, but `cc_ctx->cluster_cfg` is only assigned *after* `xcecd_start()` returns (`src/worker/src/main.c:610`), while `xcecd_start()` already performs the first join via `init_client_config()` (`src/worker/src/controller_connector.c:127-158`, hardcoded `nmasters=1`) — so the initial connection ignores `--peers` entirely and only later reconnects (`checking_connection()` → `xcecd_join_cluster()`, `controller_connector.c:237-293`) use the candidate list, one candidate at a time (app-level retry, not lib-level failover).
+- **scheduler (xce_scheduler)**: `--peers` candidate loop is correctly wired before the initial join (`src/scheduler/src/main.c:124-156`, `scheduler.c:462-495`), but there is no runtime reconnect/rejoin logic anywhere in the scheduler — if the joined controller later dies, the scheduler has no path to another candidate.
+- **apiserver**: `backend.peers` (YAML) is tried once at startup via a Go-level loop (`src/frontend/apiserver/backend/clib.go:116-133`, `RegisterToBackend()`), each attempt using `nmasters=1`. No periodic/background reconnect across peers exists — HA coverage is startup-only.
+
+In all three daemons, `xce_client_config_t.nmasters` is always set to `1` per connection attempt, so libxce's own failover rotator (`client_failover.c`) never actually engages for them — each daemon hand-rolls its own (incomplete) outer retry loop instead.
+
+A separate, partially-built shared primitive already exists in libxce (`src/include/xce/cluster.h`: `xce_connect_cluster()`, `xce_reconnect_init/step/fini()`, backed by `xce_parse_peers()`/`xce_cluster_config_validate()`) — but its production connect path (`do_connect()` in `src/lib/cluster.c`) is a stub that always returns `NULL` outside unit tests, and it operates on a raw `endpoint_t*`, not the higher-level `xcecli_ctx_t` the daemons actually use for the join/request/reply protocol. No production code calls it. Whether the design phase completes/adopts this primitive or builds directly on `xce_client_config_t.nmasters` (the `libxce-failover`/`client_failover.c` path xce_test already uses) is a design decision, not a requirements one — both existing mechanisms are noted here as prior art this feature must reconcile with, not duplicate.
+
+### Background (verification findings, 2026-07-19)
+
+`xce_test`'s `masters` setting (`src/controller/testsuite/test.c:345-377`) is the reference implementation: it parses a `host:port,...` list via `xce_parse_peers()`, sets `config.nmasters = candidates_count`, fills the full `config.masters[]` array, and calls `xcecli_ctx_init()` once — so libxce's own `client_failover.c` EndpointRotator handles mid-session failover across all candidates.
+
+None of the three production daemons match this pattern; each has a different gap:
+
+- **worker (xcecd)**: `--peers` is parsed into `cluster_cfg`, but `cc_ctx->cluster_cfg` is only assigned *after* `xcecd_start()` returns (`src/worker/src/main.c:610`), while `xcecd_start()` already performs the first join via `init_client_config()` (`src/worker/src/controller_connector.c:127-158`, hardcoded `nmasters=1`) — so the initial connection ignores `--peers` entirely and only later reconnects (`checking_connection()` → `xcecd_join_cluster()`, `controller_connector.c:237-293`) use the candidate list, one candidate at a time (app-level retry, not lib-level failover).
+- **scheduler (xce_scheduler)**: `--peers` candidate loop is correctly wired before the initial join (`src/scheduler/src/main.c:124-156`, `scheduler.c:462-495`), but there is no runtime reconnect/rejoin logic anywhere in the scheduler — if the joined controller later dies, the scheduler has no path to another candidate.
+- **apiserver**: `backend.peers` (YAML) is tried once at startup via a Go-level loop (`src/frontend/apiserver/backend/clib.go:116-133`, `RegisterToBackend()`), each attempt using `nmasters=1`. No periodic/background reconnect across peers exists — HA coverage is startup-only.
+
+In all three daemons, `xce_client_config_t.nmasters` is always set to `1` per connection attempt, so libxce's own failover rotator (`client_failover.c`) never actually engages for them — each daemon hand-rolls its own (incomplete) outer retry loop instead.
+
+## Boundary Context
+
+- **In scope**: scheduler, worker, and apiserver each using their *full* configured controller-candidate list (not just the first entry) for both the initial connection and any later reconnect; automatic reconnect across candidates for all three components (today scheduler and apiserver have none); liveness-safe retry/backoff on the worker and scheduler so a slow/unreachable candidate never stalls their other liveness-critical duties; consistent candidate-list configuration syntax across all three components; operator-visible diagnostics for connect/reconnect attempts and failures; apiserver behavior toward its own API clients while its backend connection is down or being re-established.
+- **Out of scope**: `xce_test` (already the correct reference client, not a target of this feature); raising the validated 3-candidate limit; controller-side changes (peer authentication, `validate_host()`, redirect-reply population — already shipped by `libxce-failover`); TLS/certificate distribution mechanics beyond the existing CA-cert configuration; the lightweight session-based rejoin/re-attach optimization (separate backlog item); recovering worker job-state that was lost during a disconnect window (owned by the separate `worker-rejoin-resync` item — this feature restores the *connection*, not job-state content).
+- **Adjacent expectations**: depends on `libxce-failover` (shipped) for the underlying retry primitive and controller status taxonomy (redirect vs. standby vs. not-ready); does not duplicate or replace that mechanism, only ensures each component actually drives it with a complete candidate list.
+
+## Requirements
+
+### Requirement 1: Full candidate list used on initial connection
+**Objective:** As an HA cluster operator, I want scheduler, worker, and apiserver to attempt their initial controller connection using the complete configured candidate list, so that startup succeeds even when the first-listed candidate is not the current primary.
+
+#### Acceptance Criteria
+1. When a component is started with more than one controller candidate configured, the component shall attempt connection to each configured candidate, in configured order, until one accepts the join as primary.
+2. If the first attempted candidate is unreachable, the component shall proceed to the next configured candidate without operator intervention or a restart.
+3. If a configured candidate responds as standby rather than primary, the component shall treat that as informational and proceed to the next configured candidate, not as a fatal connection error.
+4. The worker component shall use the fully configured candidate list on its first connection attempt at startup, not only on subsequent reconnects.
+5. Where only one candidate (or none) is configured, the component shall behave as it does today in single-controller mode.
+6. If every configured candidate is unreachable or rejects the join at startup, then the component shall report a clear startup failure identifying which candidates were attempted.
+
+### Requirement 2: Automatic reconnect after controller failover
+**Objective:** As an HA cluster operator, I want scheduler, worker, and apiserver to automatically reconnect to a surviving controller candidate when their active controller connection is lost, so that a controller failover does not require manually restarting these daemons.
+
+#### Acceptance Criteria
+1. When a component's active connection to its controller is lost, the component shall attempt to reconnect using the complete configured candidate list, not only the previously connected candidate.
+2. While reconnecting, the component shall keep retrying across the full configured candidate list until one candidate accepts the connection as primary or the component is shut down by an operator.
+3. If a reconnect sweep exhausts all configured candidates without success, then the component shall continue retrying rather than permanently giving up, so the component self-heals once a primary becomes reachable again.
+4. When apiserver reconnects to a new primary after a controller failover, apiserver shall resume receiving backend push notifications without requiring an apiserver process restart.
+
+### Requirement 3: Reconnect retry does not block other liveness-critical work
+**Objective:** As an HA cluster operator, I want the worker and scheduler's controller-reconnect retries to never block their other liveness-critical processing, so that a slow or unreachable controller candidate does not make the daemon itself appear unresponsive.
+
+#### Acceptance Criteria
+1. While a worker or scheduler is retrying a controller connection (including any backoff delay between attempts), the component shall continue performing its other liveness-critical duties (worker: hardware/process monitoring and job-state tracking; scheduler: its allocation/decision processing) without being blocked by the retry attempt or backoff delay.
+2. The worker and scheduler's reconnect retries shall not delay or prevent the component's own heartbeat responses to a connected controller, so a stalled reconnect attempt elsewhere cannot itself trigger an unnecessary failover.
+
+### Requirement 4: apiserver behavior toward API clients during backend disconnection
+**Objective:** As an API client of apiserver, I want a clear, immediate response when the backend controller connection is down, so that I am not left waiting indefinitely for requests that cannot currently be served.
+
+#### Acceptance Criteria
+1. When apiserver's connection to the controller is down and a reconnect is in progress, apiserver shall respond to API requests that require the backend with a clear service-unavailable style error rather than hanging until a connection is restored.
+2. While the backend reconnect is in progress, apiserver shall continue attempting to serve API requests that do not require the backend connection.
+
+### Requirement 5: Operator-visible connect/reconnect diagnostics
+**Objective:** As an HA cluster operator, I want visibility into which controller candidates each component tried and why each attempt failed, so that I can distinguish a real outage from a configuration mistake.
+
+#### Acceptance Criteria
+1. When a component attempts a connection or reconnection to a controller candidate, the component shall log the candidate attempted and the outcome (success, unreachable, or standby-redirect).
+2. If a component exhausts all configured candidates during a connection or reconnection attempt, then the component shall log a summary identifying every candidate attempted and the reason each failed.
+
+### Requirement 6: Consistent candidate-list configuration across components
+**Objective:** As an HA cluster operator, I want to configure the controller candidate list the same way for scheduler, worker, and apiserver, so that I do not need to learn or maintain three different configuration formats for the same setting.
+
+#### Acceptance Criteria
+1. The candidate-list configuration accepted by apiserver shall use the same `[name=]host:port[,...]` syntax already accepted by scheduler and worker.
+2. If a component's candidate-list configuration is malformed, then the component shall reject it with an error identifying the malformed entry, consistent with how the other components report malformed candidate-list configuration.

.kiro/specs/component-failover-adoption/research.md

@@ -0,0 +1,120 @@
+# Research & Design Decisions
+
+## Summary
+- **Feature**: `component-failover-adoption`
+- **Discovery Scope**: Extension (light discovery — libxce, worker, scheduler, apiserver already exist; this wires three consumers onto an existing library mechanism)
+- **Key Findings**:
+  - `libxce-failover` (shipped, GO 2026-07-18) already built a **complete, transparent, per-call multi-master retry/rotate mechanism** (`XCECLI_FAILOVER_CALL` / `XCECLI_FAILOVER_CALL_COUNT` / `XCECLI_FF_CALL` in `client_internal.h`, wired into every public `xcecli_*` call in `client.c`, including `xcecli_join` and the initial `xcecli_start` connect). It requires only `xce_client_config_t.nmasters > 1` to activate. **No production code sets `nmasters` above 1.**
+  - A **rejoin hook** (`xcecli_set_reconnect_handler`) exists specifically so a consumer can re-send its join after a transparent reconnect. It has **zero production callers** — only its own setter and a unit test reference it.
+  - A second, older, **half-built and unused** primitive (`xce_connect_cluster()` / `xce_reconnect_init/step/fini()` in `src/lib/cluster.c`) also exists. Its production connect path (`do_connect()`) is a stub that always returns `NULL` outside unit tests, and it operates on raw `endpoint_t*`, not `xcecli_ctx_t`. No production code calls it. It predates/overlaps `libxce-failover` and is superseded by it.
+  - This is a genuine "wire it up" feature, not a "build failover" feature — confirmed by the roadmap: `component-failover-adoption` (roadmap.md:64) was already recorded 2026-06-16 with this exact framing, HARD-blocked on `libxce-failover`, now unblocked.
+
+## Research Log
+
+### Why do worker/scheduler/apiserver each hand-roll a candidate loop instead of using the library mechanism?
+- **Context**: Initial verification (this conversation) found worker, scheduler, and apiserver each independently loop over configured candidates, calling `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` fresh for each candidate with `nmasters` hardcoded to `1` per attempt.
+- **Sources Consulted**: `src/lib/client.c`, `src/lib/client_internal.h`, `src/worker/src/controller_connector.c`, `src/scheduler/src/scheduler.c`, `src/frontend/apiserver/backend/clib.go`, `roadmap.md`.
+- **Findings**: `xcecli_start` (client.c:125-188) already rotates through ALL configured masters on the very first connect (bug-854 fix, comment at line 136). `xcecli_join` (client.c:291-300) is wrapped in `XCECLI_FAILOVER_CALL`, so it already retries and rotates on standby/redirect/transport-error responses. The three daemons' hand-rolled loops predate this (or were never updated after it landed) and now duplicate — imperfectly — what the library does correctly.
+- **Implications**: The fix is subtractive as much as additive: delete the per-daemon candidate loops, populate `nmasters`/`masters[]`/`master_ports[]` fully once, and let the existing library mechanism do the rest. This directly satisfies Requirements 1 and most of Requirement 2 without new retry/rotation logic.
+
+### Does the library mechanism, once enabled, also cover runtime reconnect (Requirement 2) automatically?
+- **Findings**: Yes for any component whose normal operation already calls `xcecli_*` functions periodically or on demand — every one of those calls is wrapped in `XCECLI_FAILOVER_CALL`/`XCECLI_FAILOVER_CALL_COUNT`, so a transport failure on any call transparently triggers rotate+reconnect+retry within that same call. Fire-and-forget calls (`xcecli_nop`, `xcecli_heartbeat`) use `XCECLI_FF_CALL` instead: a bounded single best-effort reconnect attempt, never the full retry budget — this is the library's own liveness-safety design for exactly the concern in Requirement 3.
+- **Implications**: Scheduler's `xcecli_jobdispatch` calls (scheduler.c:1289, called from the `sched_allocator` thread) and apiserver's per-request backend calls (`gCtx.ctx` in clib.go) get transparent reconnect for free once `nmasters` is set correctly — no new polling/reconnect loop needs to be written for either component.
+
+### Does anything need a NEW re-join after a transparent reconnect, beyond what the library does automatically?
+- **Findings**: The library's rejoin hook exists precisely because reconnecting the transport does not re-establish application-level cluster membership — a new primary does not automatically know a worker/scheduler/apiserver rejoined unless it re-sends its join (role) message. `xcecli_reconnect_cb` "runs on the calling thread" and returns 0 to resume the interrupted op or <0 to treat the rejoin as failed (loop rotates/backs off again).
+- **Implications**: All three components must register a rejoin callback that re-issues `xcecli_join(ctx, <role>, ...)` (with worker also re-sending `hostattrs_t`). Without it, a component would reconnect at the transport level but the new primary would not recognize it as joined — likely surfacing as silently-dropped pushes/notifications rather than an obvious error.
+
+### Thread-affinity of blocking retry calls (Requirement 3)
+- **Findings**: Worker already isolates all controller-connection work (`controller_connector_start`, `xcecd_join_cluster`, and the periodic `checking_connection()` health-check) onto one dedicated thread (`start_controller_connector_thread` / `connector_to_controller`), separate from `hardware_monitor`/`system_monitor`/`process_manager` update calls in `xcecd_start()`. This isolation already satisfies Requirement 3 structurally for the worker — it is not something this feature needs to newly build, only preserve while removing the redundant candidate loop.
+- Scheduler already runs `sched_job_picker` and `sched_allocator` as two separate `pthread_create`d threads (scheduler.c:511,516); `xcecli_jobdispatch` (the only call site found that could block on retry, scheduler.c:1289) executes from the allocator thread only. A blocked allocator during a controller outage is a benign, expected degradation (there is no work the allocator can do without a controller anyway) and does not stall the job picker or any liveness signal scheduler sends elsewhere — scheduler has no independent heartbeat-carrying thread to protect. No new threading is needed for the scheduler.
+- apiserver: unlike the C daemons, ALL request-handling goroutines that touch the backend share the same `gCtx.ctx`, serialized by the library's per-context `failover.lock`. Under a real outage this could queue every concurrent API request behind the full retry budget, which is a client-experience problem (Requirement 4), not a cluster-liveness problem. Needs a lightweight fail-fast guard in the Go layer — see Design Decisions.
+- **Implications**: No new dedicated thread is required for worker or scheduler (existing isolation already suffices — flagged as a Validation Hook to confirm during implementation, not re-derived here). apiserver needs a new, small piece of state (not a thread) to short-circuit blocking calls when the backend is known-down.
+
+### Configuration surface mismatch: `N_MAX_MASTERS` vs `MAX_HA_CANDIDATES`
+- **Findings**: `src/include/client.h:28` defines `N_MAX_MASTERS (3)` — the hard cap on `xce_client_config_t.masters[]`. `src/include/config.h:78` defines `MAX_HA_CANDIDATES 4` — the cap on `xce_cluster_config_t.candidates[]` used by `xce_parse_peers`. These are different constants. `xce_test` (`test.c:360-367`) already handles the mismatch correctly: parse up to 4 candidates, then clamp to `N_MAX_MASTERS` (3) with a `WARN` if truncated.
+- **Implications**: Any new shared conversion helper must replicate this exact clamp-and-warn behavior (not silently drop entries, not error out) so all components behave identically when 4 candidates are configured. Reconciling the two constants into one is out of scope (pre-existing, unrelated to this feature).
+
+### TLS/CA-cert wiring gap (pre-existing, out of boundary)
+- **Findings**: `xce_client_config_t` (the struct actually consumed by `xcecli_ctx_init`/`xcecli_start`, i.e. the real connection path) has no CA-cert field at all — only `nmasters`/`masters`/`master_ports`/`from`/`key`. The `ca_cert_path` that worker/scheduler's `--ca-cert` flag populates on `xce_cluster_config_t` is validated by `xce_cluster_config_validate()` but is only ever read by the orphaned `xce_connect_cluster()` stub, never by the real connect path.
+- **Implications**: This is a pre-existing gap unrelated to candidate-list wiring — the real connection already establishes TLS today (single-master) via whatever global SSL context setup exists outside this struct. Not this feature's concern; noted here so it isn't mistaken for something this design should fix.
+
+## Architecture Pattern Evaluation
+
+| Option | Description | Strengths | Risks / Limitations | Notes |
+|--------|-------------|-----------|---------------------|-------|
+| Adopt existing `xce_client_config_t` / `XCECLI_FAILOVER_CALL` mechanism (chosen) | Populate the full candidate list once per component, register a rejoin callback, delete hand-rolled loops | Already built, tested (`libxce-failover`), and exactly what `xce_test` uses today; minimal new code | None material — this is the reference-verified path | Matches roadmap's own framing of this backlog item |
+| Complete and adopt `xce_connect_cluster()`/`xce_reconnect_*` | Finish the stubbed `do_connect()`, wire it under each daemon | N/A | Duplicate mechanism to the one already shipped; raw-`endpoint_t` API doesn't carry the join/request/reply protocol the daemons need — would need its own substantial new integration layer on top | Rejected — would build a second, redundant failover mechanism |
+| Build a new per-component reconnect loop | Fix each daemon's existing hand-rolled loop in place, independently | Minimal library changes | Perpetuates 3 divergent implementations (the root problem this feature exists to fix); does not gain the library's already-tested retry/backoff/classification logic | Rejected — directly contradicts the roadmap item's stated goal |
+
+## Design Decisions
+
+### Decision: Adopt the existing `xcecli_ctx_t` / `XCECLI_FAILOVER_CALL` mechanism; do not build new failover logic
+- **Context**: Requirements 1-3 need multi-candidate connect + runtime reconnect + liveness-safe retry.
+- **Alternatives Considered**: see Architecture Pattern Evaluation above.
+- **Selected Approach**: Each component populates a fully-formed `xce_client_config_t` (all configured candidates, not just the first) once, calls `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` once, and registers a rejoin callback via `xcecli_set_reconnect_handler`. All subsequent `xcecli_*` calls already retry/rotate/reconnect transparently.
+- **Rationale**: The mechanism is shipped, tested, and is the exact one `xce_test`'s `masters` setting already uses correctly. Building anything new would duplicate it.
+- **Trade-offs**: None material for worker/scheduler. For apiserver, moving off the current Go-level per-peer loop means the CGO layer becomes responsible for candidate rotation instead of Go — acceptable since it is strictly more capable (transparent runtime reconnect, not just startup).
+- **Follow-up**: Confirm during implementation that worker's hardware/process-monitor update calls truly run off the connector thread (Validation Hook, not re-verified byte-for-byte in this research pass).
+
+### Decision: One new shared C helper converts `xce_cluster_config_t` → `xce_client_config_t`
+- **Context**: All three components need to turn a parsed/validated candidate list into the `nmasters`/`masters[]`/`master_ports[]` triple, replicating `xce_test`'s clamp-to-`N_MAX_MASTERS`-with-WARN behavior (see mismatch finding above). Today this logic exists ad hoc in `test.c` only.
+- **Alternatives Considered**: (1) Duplicate the clamp/copy logic in worker, scheduler, and apiserver's CGO layer separately. (2) One shared helper in `src/lib/cluster.c` (where `xce_parse_peers`/`xce_cluster_config_validate` already live), declared in `xce/cluster.h`.
+- **Selected Approach**: (2) — a single `xce_cluster_config_to_client_config()` helper, reused by worker (C), scheduler (C), and apiserver (via CGO).
+- **Rationale**: DRY; the clamp-and-warn behavior must be identical across components (Requirement 6's config-syntax-parity spirit extends naturally to the derived struct, not just the input string); one function is one thing to test.
+- **Trade-offs**: None — this is pure generalization of logic that would otherwise be copy-pasted three times.
+
+### Decision: apiserver calls the existing C `xce_parse_peers`/`xce_cluster_config_validate` via CGO instead of maintaining a second Go parser
+- **Context**: Requirement 6 requires apiserver's candidate-list syntax to match scheduler/worker exactly. apiserver currently has its own hand-written Go `parsePeer` function.
+- **Alternatives Considered**: (1) Keep/harden the Go parser to match C syntax by inspection. (2) Call the real C parser via CGO (apiserver already links libxce via CGO for everything else).
+- **Selected Approach**: (2).
+- **Rationale**: Guarantees byte-identical parsing forever (no drift risk), and apiserver already pays the CGO cost for every other backend interaction.
+- **Trade-offs**: One more CGO surface, consistent with the existing `backend/` package pattern.
+
+### Decision: apiserver gets a small Go-side "backend health" circuit breaker for Requirement 4
+- **Context**: Requirement 4 needs apiserver to fail fast toward its own HTTP clients when the backend is down, rather than queue every request behind the library's full retry budget.
+- **Alternatives Considered**: (1) Let every request block on the existing per-ctx lock and rely on Gin/client-side timeouts. (2) Track backend health from call outcomes + the rejoin callback, and check it before issuing a blocking backend call.
+- **Selected Approach**: (2) — new `backend/health.go`: an atomic "degraded since" state, set when a call's classified outcome is retry-budget-exhaustion, cleared when the rejoin callback fires (successful reconnect+rejoin). Backend-call wrappers in `clib.go` check it first and return a clear error immediately while degraded, instead of entering the blocking call.
+- **Rationale**: Small, localized, reuses signals the library already produces (call rc, rejoin callback) rather than adding new polling.
+- **Trade-offs**: A short window exists where the flag is stale (a request arrives just as the backend goes down, before any call has failed) — acceptable, matches typical circuit-breaker behavior; the request in flight simply experiences the one blocking retry cycle before the flag is set for subsequent requests.
+
+### Decision: rejoin callbacks must be lock-free flag-setters, never direct `xcecli_join` calls
+- **Context**: A second `/kiro-validate-design` pass, cross-checking the library-invoked rejoin path against `client_failover.c` directly rather than only its header comments, found that `XCECLI_FAILOVER_CALL` holds `ac->failover.lock` — a plain, non-recursive `pthread_mutex_t` (`mylock_init` → `pthread_mutex_init(l, NULL)`) — for its entire retry loop, and `xcecli_invoke_rejoin()` (which calls the registered rejoin callback) runs synchronously *inside* that loop, lock held (`client_failover.c:397`). Every component's original design had its rejoin callback call `xcecli_join(...)` directly — itself `XCECLI_FAILOVER_CALL`-wrapped, so it would try to re-lock the same mutex on the same thread: a guaranteed self-deadlock on the very first real failover. This was never caught because `xcecli_set_reconnect_handler` has zero production callers and the existing unit test's callback never calls back into the client (it's a no-op probe).
+- **Alternatives Considered**: (1) Have the callback call `xcecli_join` directly (original design — deadlocks). (2) Expose a new, lock-free/raw join primitive from libxce for rejoin callbacks to use (would require modifying `libxce-failover`, violating this spec's Allowed Dependencies boundary of consuming it read-only). (3) Callback only sets a lightweight flag and returns immediately (no I/O, no lock); the daemon's own calling code checks and clears the flag *after* the triggering call returns (lock free by then) and issues the real `xcecli_join` as a separate call.
+- **Selected Approach**: (3), applied uniformly to worker (`checking_connection`), scheduler (the allocator, after `xcecli_jobdispatch`), and apiserver (each backend-call wrapper, after its `C.xcecli_*` call). apiserver's heartbeat-triggered path (Task 4.4) was already structured this way by construction (it acts after `xcecli_heartbeat` returns) and needed no change.
+- **Rationale**: Fixes the deadlock without touching `libxce` at all, staying inside this spec's stated boundary; the fix is a small, uniform pattern applied identically across all three components rather than a special case.
+- **Trade-offs**: One extra check-and-clear step after each relevant call site, in exchange for correctness; negligible cost.
+- **Follow-up**: Worth flagging upstream to `libxce-failover`'s owners that the rejoin-hook contract (`client.h`'s doc comment for `xcecli_reconnect_cb`) doesn't document this lock-held constraint — a future consumer without this spec's context could reintroduce the deadlock. Not fixed here (out of boundary), but the regression-guard unit test added in design.md's Testing Strategy exists specifically because of this gap in the upstream contract's documentation.
+
+### Decision: apiserver's rejoin registration uses a C shim in `clib.go`, not a standalone `reconnect.go` export
+- **Context**: `/kiro-validate-design` review flagged that design.md originally described `reconnect.go` as "a `//export`ed Go function registered via `xcecli_set_reconnect_handler`," implying direct registration. This codebase already documents (`wire_dispatch_support.go:154-159`, explaining why `go_push_handler` needs the `xce_backend_handler` C shim) that cgo symbol resolution is per-file — a `C.<symbol>` reference only resolves in the file holding its own matching `extern` declaration.
+- **Alternatives Considered**: (1) Register the exported Go function directly from `reconnect.go`. (2) Add a static C shim + `extern` declaration in `clib.go`'s own cgo preamble (mirroring the proven `xce_backend_handler` pattern) and register that.
+- **Selected Approach**: (2).
+- **Rationale**: Matches the codebase's own documented, working pattern instead of risking the same resolution failure `xce_backend_handler` was built to avoid.
+- **Trade-offs**: None — same amount of code, just correctly placed.
+- **Follow-up**: none; resolved by construction in design.md.
+
+### Decision: apiserver's heartbeat prober explicitly triggers rejoin on recovery
+- **Context**: `/kiro-validate-design` review found, by direct inspection of `client_internal.h`/`client_failover.c`, that `XCECLI_FF_CALL` (backing `xcecli_heartbeat`, apiserver's only idle-period liveness probe) never calls `xcecli_invoke_rejoin` — only `XCECLI_FAILOVER_CALL` (the full-retry path used by write calls) does. Left unaddressed, apiserver would repair its TCP connection via the heartbeat but never re-join, so push notifications (Req 2.4) would stay dead during any failover where no write request happens to arrive first.
+- **Alternatives Considered**: (1) Switch the periodic prober from `xcecli_heartbeat` (fire-and-forget) to `xcecli_hello` (full retry, rejoin-bearing) — the same mechanism worker's `checking_connection()` already uses. (2) Keep `xcecli_heartbeat` (preserves its existing wire semantics/side effects on the controller) and have the Go-side tick callback explicitly detect a failure-to-success transition and manually invoke the same rejoin logic used elsewhere.
+- **Selected Approach**: (2).
+- **Rationale**: Preserves `xcecli_heartbeat`'s existing wire behavior (unknown/unverified whether the controller treats the heartbeat opcode differently from hello for apiserver accounting) while closing the gap with a small, self-contained Go-side check — no library changes, no behavior change to any other component.
+- **Trade-offs**: A few extra lines in `heartbeat.go`; the rejoin logic now has two call sites (the cgo shim and the heartbeat callback), both invoking the same `reconnect.go` function, so no duplicated join-payload logic.
+- **Follow-up**: none; resolved by construction in design.md.
+
+### Decision: Do not touch `xce_connect_cluster()`/`xce_reconnect_*`
+- **Context**: This second, unused, stubbed primitive exists in the same area of the codebase.
+- **Selected Approach**: Leave as-is; do not adopt, complete, or remove it in this spec.
+- **Rationale**: Out of this feature's boundary — removing dead code is a separate hardening decision, not required to satisfy any requirement here, and touching it risks scope creep into an unrelated cleanup.
+- **Follow-up**: Worth a future hardening backlog note (superseded/dead code) — not filed as part of this spec.
+
+## Risks & Mitigations
+- Deleting worker's/scheduler's hand-rolled candidate loops could regress subtle existing behavior (e.g., worker's specific WARN logging on standby redirect) — mitigate by preserving equivalent log statements around the new single-call path, verified in code review.
+- apiserver's circuit breaker could mask a fixed backend as still-degraded if the clearing signal (rejoin callback) doesn't fire in some edge case — mitigate with a bounded TTL on the degraded flag (re-probe after a short timeout) rather than relying solely on the rejoin callback.
+- `N_MAX_MASTERS` (3) vs `MAX_HA_CANDIDATES` (4) mismatch could silently drop a 4th configured candidate on any component — mitigate by making the shared helper's WARN-and-clamp behavior loud and identical everywhere (already the plan).
+
+## References
+- `src/lib/client.c`, `src/lib/client_internal.h`, `src/include/xce/client_failover.h` — the shipped `libxce-failover` mechanism this feature adopts.
+- `src/controller/testsuite/test.c:345-382` — the reference (`xce_test`) usage pattern.
+- `.kiro/steering/roadmap.md:64` — `component-failover-adoption` backlog item this spec formalizes.
+- `.kiro/steering/ha-async-io.md` — thread-role contract informing Requirement 3.

.kiro/specs/component-failover-adoption/spec.json

@@ -0,0 +1,22 @@
+{
+  "feature_name": "component-failover-adoption",
+  "created_at": "2026-07-20T04:23:58Z",
+  "updated_at": "2026-07-20T05:20:00Z",
+  "language": "en",
+  "phase": "tasks-generated",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": true
+    },
+    "design": {
+      "generated": true,
+      "approved": true
+    },
+    "tasks": {
+      "generated": true,
+      "approved": false
+    }
+  },
+  "ready_for_implementation": false
+}

.kiro/specs/component-failover-adoption/tasks.md

@@ -0,0 +1,126 @@
+# Implementation Plan
+
+- [ ] 1. Foundation: shared candidate-list conversion helper
+- [ ] 1.1 Implement the candidate-list-to-client-config conversion helper in libxce
+  - Add `xce_cluster_config_to_client_config()` to `src/lib/cluster.c`, declared in `src/include/xce/cluster.h`, generalizing the clamp-to-`N_MAX_MASTERS`-with-WARN logic `xce_test` already applies inline
+  - Copies candidates in order; leaves `from`/`key` untouched for the caller to set
+  - Observable completion: given a validated `xce_cluster_config_t` with 1, 3, and 4 candidates, the function returns a correctly populated `xce_client_config_t` in each case, truncating and logging a WARN only for the 4-candidate case
+  - _Requirements: 1.5_
+
+- [ ] 1.2 Unit test the conversion helper
+  - Cover: candidate order preserved, truncation at `N_MAX_MASTERS` with WARN when given more, `NULL`/empty-candidate-list rejection with `-EINVAL`
+  - Add cases to `src/lib/testsuite/test_cluster.c`
+  - Observable completion: new test cases pass under the existing libxce unit test run
+  - _Requirements: 1.5_
+
+- [ ] 2. Worker: adopt full-candidate-list connect and transparent reconnect
+- [ ] 2.1 (P) Replace worker's per-candidate join loop with a single library-driven connect/join
+  - In `src/worker/src/controller_connector.c`: delete `xcecd_join_cluster`'s manual per-candidate iteration and `init_client_config`'s hardcoded `nmasters=1`; build the client config via the Task 1 helper from `cc_ctx->cluster_cfg` (or the existing single-host fallback when no `--peers` given) and call `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` once
+  - Register a rejoin callback via `xcecli_set_reconnect_handler` (declared in `src/worker/include/controller_connector.h`) that ONLY sets a `needs_rejoin` flag and returns `0` — it must NOT call `xcecli_join` or any other `xcecli_*` function itself, because the library invokes it while `ac->failover.lock` is still held (a non-recursive mutex) and re-entering that lock via a wrapped call self-deadlocks
+  - In `checking_connection()`, immediately after each `xcecli_hello` call returns (lock now released), check and clear `needs_rejoin`; if set, call `xcecli_join(ctx, HostRole_worker, &host_attrs)` as a separate, subsequent call
+  - Preserve existing WARN/INFO log statements for standby-redirect and per-candidate outcomes around the new single-call path
+  - Reconcile `checking_connection()`'s bounded `MAX_RETRY_COUNT`/`RETRY_INTERVAL`-then-`handle_disconnect()` loop with the library's own indefinite retry: the library now owns connection recovery (Requirement 2.3 — keep retrying, never permanently give up), so `checking_connection()`'s outer bound must no longer tear the worker down after 5 failed hello probes; it becomes a liveness-probe cadence only
+  - Observable completion: starting the worker against a 3-candidate config where candidate[0] is a live standby results in the worker joined as expected, with log lines showing the standby redirect and the successful candidate; a simulated outage longer than the old 5-retry window no longer triggers `handle_disconnect()`; invoking the registered rejoin callback while the ctx's failover lock is held returns immediately without blocking
+  - _Requirements: 1.1, 1.2, 1.3, 1.5, 1.6, 2.1, 2.2, 2.3, 5.1, 5.2_
+  - _Boundary: WorkerControllerConnector_
+  - _Depends: 1.1_
+
+- [ ] 2.2 Fix worker's candidate-list startup ordering
+  - In `src/worker/src/main.c`: move the `cc_ctx->cluster_cfg = cluster_cfg` assignment to before `xcecd_start()` is called, so the very first connection attempt (inside `xcecd_start`/`controller_connector_start`) already sees the full `--peers` candidate list instead of only the single-host fallback
+  - Observable completion: starting the worker with `--peers` where the FIRST listed candidate is unreachable succeeds on the first connect attempt (no restart needed), verified by a log trace showing candidate rotation during the initial join, not only during a later reconnect
+  - _Requirements: 1.4_
+
+- [ ] 2.3 Validate worker thread isolation and add worker candidate/reconnect test coverage
+  - Confirm (via code read) that `hardware_monitor_update`/`system_monitor_update`/`process_manager_update` do not execute on the connector thread that now carries the library's blocking retry/backoff — fix isolation if this is not already true
+  - Add `src/worker/testsuite/test_e2e_multimaster_join.c`: worker starts and joins across a multi-candidate config; killing the joined primary results in reconnect + successful re-join to a surviving candidate without a worker restart
+  - Observable completion: the new test passes, and worker's hardware/process monitoring continues updating (observable via existing monitor state) while a connect/reconnect retry is in progress
+  - _Requirements: 1.6, 3.1, 3.2_
+  - _Depends: 2.1, 2.2_
+
+- [ ] 3. Scheduler: adopt full-candidate-list connect and add runtime reconnect
+- [ ] 3.1 (P) Replace scheduler's inline per-candidate loop with a single library-driven connect/join
+  - In `src/scheduler/src/scheduler.c`: delete `sched_init`'s inline per-candidate loop; build the client config via the Task 1 helper from `sched->cluster_cfg` (or the existing single-host fallback) and call `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` once
+  - Register a rejoin callback via `xcecli_set_reconnect_handler` that ONLY sets a `needs_rejoin` flag and returns `0` — same constraint as worker's Task 2.1: the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join` itself or it self-deadlocks
+  - After each `xcecli_jobdispatch` call returns (lock now released), check and clear `needs_rejoin`; if set, call `xcecli_join(ctx, HostRole_scheduler, NULL)` as a separate, subsequent call before continuing
+  - Preserve/add per-candidate outcome logging equivalent to what the deleted loop produced
+  - Observable completion: starting the scheduler against a 3-candidate config where candidate[0] is a live standby results in the scheduler joined as expected, with log lines showing the standby redirect and the successful candidate — and killing the joined primary afterward results in automatic reconnect and re-join to a surviving candidate, which the scheduler has never done before this task; invoking the registered rejoin callback while the ctx's failover lock is held returns immediately without blocking
+  - _Requirements: 1.1, 1.2, 1.3, 1.5, 1.6, 2.1, 2.2, 2.3, 5.1, 5.2_
+  - _Boundary: SchedulerControllerClient_
+  - _Depends: 1.1_
+
+- [ ] 3.2 Add scheduler candidate/reconnect test coverage
+  - Add `src/scheduler/testsuite/test_e2e_multimaster_join.c`: scheduler starts and joins across a multi-candidate config; killing the joined primary results in reconnect and re-join without a scheduler restart
+  - Confirm `sched_job_picker` continues running on its own thread, unaffected, while `sched_allocator`'s `xcecli_jobdispatch` call is blocked in reconnect/backoff (existing thread split, not new code)
+  - Observable completion: the new test passes, and the job-picker thread's activity (e.g. its own state/counters) continues advancing during a simulated reconnect delay on the allocator thread
+  - _Requirements: 3.1, 3.2_
+  - _Depends: 3.1_
+
+- [ ] 4. apiserver: adopt full-candidate-list connect via CGO, config parity, rejoin, and fail-fast
+- [ ] 4.1 (P) Route apiserver's candidate-list parsing through the real C parser
+  - In `src/frontend/apiserver/backend/clib.go`: call `xce_parse_peers`/`xce_cluster_config_validate` via CGO to parse `config.Peers` into a `xce_cluster_config_t`, replacing the hand-written Go `parsePeer` function and its call sites
+  - Map C validation errors (`-EINVAL` cases) to clear Go error messages naming the malformed entry
+  - Observable completion: the same malformed-candidate-string test cases that scheduler/worker already reject (from Task 1.2's neighboring validation coverage) are rejected identically when passed through apiserver's config path
+  - _Requirements: 6.1, 6.2_
+  - _Boundary: ApiserverBackendBridge_
+  - _Depends: 1.1_
+
+- [ ] 4.2 Replace apiserver's per-peer connect loop with a single library-driven connect
+  - In `src/frontend/apiserver/backend/clib.go`: replace `RegisterToBackend`'s `for _, peer := range config.Peers { tryConnect(...) }` loop with one call that builds the client config via the Task 1 helper (through CGO) from the full parsed candidate list (or the single `Address`/`Port` fallback) and calls `xce_backend_init` once
+  - Preserve/extend existing connect-outcome logging (`zap`) to report each candidate attempted
+  - Observable completion: starting apiserver with 3 configured peers where the first is a live standby results in apiserver registered against the actual primary in a single `RegisterToBackend` call, with log lines showing the standby redirect
+  - _Requirements: 1.1, 1.2, 1.3, 1.5, 1.6, 5.1, 5.2_
+  - _Depends: 4.1_
+
+- [ ] 4.3 Add apiserver's rejoin logic with a correctly-placed cgo registration
+  - New `src/frontend/apiserver/backend/reconnect.go`: two plain Go functions — (1) a `//export`-annotated flag-setter that ONLY sets a `needs_rejoin` atomic flag and returns `0` (no I/O — the library invokes it while `ac->failover.lock` is held, so calling `xcecli_join` directly here self-deadlocks); (2) a plain function that performs the actual rejoin — calls `xcecli_join(gCtx.ctx, HostRole_apiserver, NULL)` and clears backend-health degraded state on success
+  - In `src/frontend/apiserver/backend/clib.go`'s cgo preamble: add a static C shim (mirroring the existing `xce_backend_handler`/`go_push_handler` pattern) with its own `extern` declaration for the flag-setter export, and pass the shim to `xcecli_set_reconnect_handler` — the registration must live in `clib.go`, not in `reconnect.go`, because cgo symbol resolution is per-file (same reason `go_push_handler` needs `xce_backend_handler`, documented in `wire_dispatch_support.go`)
+  - Add one shared internal helper in `clib.go` that wraps a `C.xcecli_*` call: after the call returns (lock now released), check and clear `needs_rejoin`; if set, call `reconnect.go`'s actual-rejoin function as a separate step. Route all four backend-call wrappers (job submit, job control, queue control, host control) through this ONE helper instead of duplicating the check in each (Task 4.5 extends this same helper with the pre-call health check, rather than adding a second one)
+  - Observable completion: after killing and restoring connectivity to apiserver's joined controller while a write request is in flight (job submit retried through the full failover path), apiserver resumes receiving backend push notifications (existing `go_push_handler` path) without an apiserver process restart; invoking the registered flag-setter while the ctx's failover lock is held returns immediately without blocking
+  - _Requirements: 2.1, 2.2, 2.3, 2.4_
+  - _Depends: 4.2_
+
+- [ ] 4.4 Trigger apiserver's rejoin from the idle-period heartbeat path
+  - In `src/frontend/apiserver/backend/heartbeat.go`'s tick callback: track the previous tick's `xcecli_heartbeat` outcome; when a tick transitions from failure to success (the fire-and-forget call's own best-effort transport reconnect just succeeded), explicitly call Task 4.3's actual-rejoin function (function (2), not the flag-setter) directly — safe here since `XCECLI_FF_CALL` has already released the lock by the time the heartbeat call returns. `XCECLI_FF_CALL` (backing `xcecli_heartbeat`) never invokes the library's own rejoin hook, unlike the full-retry path write calls use, so without this the idle-period case never re-joins
+  - Observable completion: killing the primary while apiserver has no in-flight write request (only the heartbeat ticking) still results in apiserver rejoining and resuming push notifications, verified by the same push-notification check as Task 4.3 but with no write traffic present during the outage
+  - _Requirements: 2.4_
+  - _Depends: 4.3_
+
+- [ ] 4.5 Add apiserver's backend-health circuit breaker
+  - New `src/frontend/apiserver/backend/health.go`: atomic degraded-until state with `IsDegraded()`, `MarkDegraded(ttl)`, `MarkHealthy()`
+  - Extend Task 4.3's shared internal helper (not a second, separate hook) to also check `IsDegraded()` before the `C.xcecli_*` call and return a clear "backend unavailable" error immediately instead of invoking it when degraded; wire `MarkDegraded` into the same helper's post-call path when a call's result indicates retry-budget exhaustion; wire `MarkHealthy` into both Task 4.3's rejoin function (write-path recovery) and Task 4.4's heartbeat-triggered rejoin (idle-path recovery)
+  - Observable completion: with the flag forced degraded in a unit test, a wrapped backend call returns immediately with the unavailable error instead of invoking the C call
+  - _Requirements: 4.1, 4.2_
+  - _Depends: 4.4_
+
+- [ ] 4.6 apiserver unit test coverage
+  - `src/frontend/apiserver/backend/health_test.go`: degraded→healthy via `MarkHealthy()` and via TTL expiry; concurrent-goroutine safety
+  - Extend `src/frontend/apiserver/backend/clib_test.go` with candidate-list parity cases from Task 4.1
+  - Observable completion: `make test` in `src/frontend/` passes with the new cases included
+  - _Requirements: 4.1, 4.2, 6.1, 6.2_
+  - _Depends: 4.5_
+
+- [ ] 5. Integration and cross-component validation
+- [ ] 5.1 E2E: full-cluster reconnect after primary failover
+  - Extend the project's existing `/xce-deploy-ha` + `/xce-test-ha` pattern to a scenario with worker, scheduler, and apiserver all configured with the full 3-candidate list; kill the primary mid-session with a write request in flight
+  - Assert: worker reconnects and resumes job-state reporting; scheduler reconnects and resumes dispatch; apiserver reconnects, resumes serving API requests, and resumes receiving push notifications — none of the three restarted
+  - Observable completion: the soak scenario passes with all three components showing a successful re-join to the new primary in their logs and no manual intervention
+  - _Requirements: 2.1, 2.2, 2.3, 2.4_
+  - _Depends: 2.3, 3.2, 4.6_
+
+- [ ] 5.2 E2E: apiserver reconnects during an idle period (no in-flight write traffic)
+  - Kill the primary while apiserver has only its heartbeat prober active (no pending write requests); assert apiserver still rejoins and resumes push notifications via the Task 4.4 heartbeat-triggered path
+  - Observable completion: the scenario passes without requiring any client-triggered write request during the outage window
+  - _Requirements: 2.4_
+  - _Depends: 4.6_
+
+- [ ] 5.3 E2E: apiserver fails fast while backend is degraded
+  - With apiserver's backend connection killed and not yet reconnected, issue an API request that requires the backend and assert it returns a clear 503-style error within a bounded, sub-second time — not after the full retry budget
+  - Observable completion: the timed assertion passes against a live apiserver instance
+  - _Requirements: 4.1, 4.2_
+  - _Depends: 4.6_
+
+- [ ] 5.4 Regression: confirm existing single-controller mode is unaffected
+  - Run the full worker, scheduler, and apiserver test suites (ASAN where applicable) plus the existing non-HA (no `--peers`/no `backend.peers`) startup path for each component
+  - Observable completion: all existing tests pass unchanged, and each component still starts and operates normally when no candidate list is configured (Requirement 1.5's single/no-candidate fallback)
+  - _Requirements: 1.5_
+  - _Depends: 5.1, 5.2, 5.3_