myxce/.kiro/specs/component-failover-adoption/design.md

# Design Document: component-failover-adoption

## Overview

Scheduler, worker, and apiserver each connect to the controller cluster today using only a single configured endpoint at a time, with divergent, partly-broken hand-rolled logic for trying alternates. Meanwhile `libxce` already ships a complete, tested, transparent multi-master mechanism (`XCECLI_FAILOVER_CALL` and friends in `src/lib/client.c`) — the same one `xce_test`'s `masters` setting already uses correctly. This design wires all three components onto that existing mechanism: populate the full candidate list once, let the library rotate/retry/reconnect on every call, and register a rejoin callback so a component re-establishes cluster membership after a transparent reconnect.

**Users**: HA cluster operators who configure multiple controller candidates (`--peers` for scheduler/worker, `backend.peers` YAML for apiserver) and expect scheduler, worker, and apiserver to survive a controller failover without manual restarts — the same expectation `xce_test` already meets.

**Impact**: Removes worker's and scheduler's hand-rolled per-candidate join loops (deleting, not adding, most of the connection-retry code in `controller_connector.c` and `scheduler.c`); replaces apiserver's Go-level per-peer loop with a single library-driven connect; adds one new small shared C helper and a small Go-side health guard. No changes to the controller or to `libxce`'s failover mechanism itself.

### Goals
- Scheduler, worker, and apiserver each hand the full configured candidate list to `xce_client_config_t` on the very first connection attempt (Req 1).
- All three automatically reconnect and resume cluster membership after their controller connection is lost, without a process restart (Req 2).
- Worker's and scheduler's retry/backoff never blocks duties independent of controller connectivity (Req 3).
- apiserver returns a clear, fast error to its own API clients while its backend connection is down, instead of hanging (Req 4).
- Every connect/reconnect attempt and outcome is logged per-candidate (Req 5).
- apiserver accepts the identical candidate-list syntax scheduler/worker already accept (Req 6).

### Non-Goals
- Changing `xce_test` (it is already the reference-correct client).
- Completing or adopting the separate, unused `xce_connect_cluster()`/`xce_reconnect_*` primitive in `cluster.c` — superseded by the mechanism this design adopts; left untouched.
- Raising the `N_MAX_MASTERS` (3) cap or reconciling it with `MAX_HA_CANDIDATES` (4) — pre-existing constraint, replicated (clamp + WARN) exactly as `xce_test` already does.
- Wiring CA-cert/TLS trust into the real connect path — `xce_client_config_t` carries no such field today and this feature does not add one; TLS is established the same way it already is.
- Controller-side changes (redirect-reply population, peer authentication) — already shipped by `libxce-failover`.
- Recovering worker job-state lost during a disconnect window — owned by `worker-rejoin-resync`.

## Boundary Commitments

### This Spec Owns
- A new shared libxce helper that converts a validated `xce_cluster_config_t` (parsed candidate list) into a fully-populated `xce_client_config_t`.
- Worker's, scheduler's, and apiserver's integration code that calls `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` with the full candidate list and registers a rejoin callback.
- apiserver's candidate-list config parsing (routed through the existing C parser via CGO instead of a second Go implementation).
- apiserver's client-facing fail-fast behavior while its backend connection is degraded.
- Connect/reconnect diagnostic logging in all three components.

### Out of Boundary
- `xce_test` and its `masters` setting (unchanged, reference implementation).
- `libxce`'s failover mechanism itself (`client.c`, `client_internal.h`, `client_failover.c`) — consumed as-is, not modified.
- `xce_connect_cluster()`/`xce_reconnect_*` (unused, untouched).
- Controller-side membership/redirect logic.
- Worker job-state resync content on rejoin (`worker-rejoin-resync`).
- `N_MAX_MASTERS`/`MAX_HA_CANDIDATES` value changes.
- TLS/CA-cert plumbing into the real connect path.

### Allowed Dependencies
- `src/lib/client.c` / `client_internal.h` / `client_failover.h` (shipped `libxce-failover` mechanism) — read-only dependency, no modification.
- `src/lib/cluster.c` / `xce/cluster.h` (`xce_parse_peers`, `xce_cluster_config_validate`) — existing, unmodified; this spec adds one new function alongside them.
- Existing per-component candidate-list CLI/YAML surfaces (`--peers`, `--ca-cert`, `backend.peers`) — unmodified externally; only how they're consumed changes.

### Revalidation Triggers
- Any change to `xce_client_config_t`'s layout or `N_MAX_MASTERS`.
- Any change to `XCECLI_FAILOVER_CALL`/`XCECLI_FF_CALL` retry/classification semantics (`libxce-failover`).
- Any change to `xce_parse_peers`/`xce_cluster_config_validate` syntax or validation rules (apiserver's CGO-based parser would need re-verification).
- `worker-rejoin-resync` landing — worker's rejoin callback becomes the natural place to also trigger job-state resync; this spec's callback is a no-op-beyond-rejoin placeholder for that.

## Architecture

### Existing Architecture Analysis
Today each of the three components independently drives its own `xcecli_ctx_t` lifecycle with `nmasters` hardcoded to `1` per connection attempt, and layers its own outer candidate loop on top:
- **worker**: `controller_connector.c`'s `xcecd_join_cluster` loops candidates itself, but the very first connection (inside `xcecd_start()`) happens before `cc_ctx->cluster_cfg` is even populated (`main.c` assigns it after `xcecd_start()` returns) — the candidate list is silently ignored on startup.
- **scheduler**: `scheduler.c`'s `sched_init` loops candidates correctly at startup, but there is no reconnect logic anywhere afterward.
- **apiserver**: `clib.go`'s `RegisterToBackend` loops candidates once via a hand-written Go parser, only at process startup.

None of the three ever set `nmasters > 1` on a single `xce_client_config_t`, so the library's own rotation/retry (`client.c`, wrapped by `XCECLI_FAILOVER_CALL`) never actually engages for them.

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    subgraph Daemons
        Worker[worker xcecd]
        Scheduler[xce_scheduler]
        Apiserver[apiserver backend package]
    end

    subgraph NewSharedCode
        Convert[xce_cluster_config_to_client_config]
    end

    subgraph ExistingLibxce
        Parse[xce_parse_peers]
        Validate[xce_cluster_config_validate]
        CtxInit[xcecli_ctx_init xcecli_start xcecli_join]
        Failover[XCECLI_FAILOVER_CALL rotate retry backoff]
        Rejoin[rejoin callback xcecli_set_reconnect_handler]
    end

    subgraph Cluster
        Primary[Controller Primary]
        Standby1[Controller Standby]
    end

    Worker --> Parse
    Scheduler --> Parse
    Apiserver --> Parse
    Parse --> Validate
    Validate --> Convert
    Convert --> CtxInit
    CtxInit --> Failover
    Failover --> Rejoin
    Failover --> Primary
    Failover --> Standby1
```

**Architecture Integration**:
- Selected pattern: thin per-component integration layer over an unchanged, already-shipped library mechanism (adapter, not new subsystem).
- Domain boundaries: `libxce` owns candidate parsing, validation, connect/rotate/retry/backoff, and rejoin-hook invocation (unchanged). Each daemon owns only: building the candidate list from its own config surface, calling the library once, and supplying its role-specific rejoin behavior (what to send when re-joining).
- Existing patterns preserved: the `xce_parse_peers`/`xce_cluster_config_validate`/`xce_client_config_t` split; the CGO bridge pattern already used throughout `backend/`; worker's dedicated connector thread.
- New components: one C helper (candidate-list → client-config conversion, generalized out of `xce_test`'s inline logic) and one small Go health-state module (apiserver-only, has no C-side equivalent need).
- Steering compliance: no new blocking work introduced on iomgr/delaymgr-equivalent threads; worker's existing thread isolation is preserved, not altered.

### Rejoin callback constraint: avoiding self-deadlock on `ac->failover.lock`

Verified directly against `client_failover.c:341-411`: `XCECLI_FAILOVER_CALL` holds `ac->failover.lock` (a plain, non-recursive `pthread_mutex_t` — `mylock_init` calls `pthread_mutex_init(l, NULL)`, no `PTHREAD_MUTEX_RECURSIVE` attribute) for the **entire** retry loop, and `xcecli_invoke_rejoin()` — which calls the registered rejoin callback — is invoked **synchronously from inside that loop, while the lock is held** (`client_failover.c:397`). Any rejoin callback that itself calls a public `xcecli_*` wrapper on the *same* `xcecli_ctx_t` (e.g. `xcecli_join`, which is itself `XCECLI_FAILOVER_CALL`-wrapped and tries to re-lock the same mutex) **self-deadlocks** — the calling thread blocks forever on its own already-held lock. This was never caught because `xcecli_set_reconnect_handler` has zero production callers today and the existing unit test's callback is a no-op probe that never calls back into the client.

This directly constrains every component's rejoin implementation in this spec:
- The callback **registered with the library** (worker's `checking_connection`→`xcecli_hello` path, scheduler's `xcecli_jobdispatch` path, apiserver's write-call path) must do **no I/O and call no `xcecli_*` function** — it may only set a lightweight "needs rejoin" flag (a plain bool/atomic on the consumer side) and return `0` so the interrupted operation resumes normally.
- Each component's own calling code, **after** the triggering `xcecli_*` call returns (at which point the library has released the lock), checks and clears that flag and issues `xcecli_join(...)` **as a separate, subsequent call** — safe, since the lock is free by then.
- apiserver's heartbeat-triggered path (see below) is unaffected by this constraint — it already detects recovery and calls the rejoin logic **after** `xcecli_heartbeat` has returned (`XCECLI_FF_CALL` also releases its lock before returning), so it was never at risk.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|------------------|-------|
| Shared library | C (gnu17), `src/lib/cluster.c` | New candidate→client-config conversion helper | Extends existing file alongside `xce_parse_peers` |
| Worker / Scheduler | C (gnu17) | Consume the existing `xcecli_ctx_t` mechanism directly | No new dependency |
| apiserver | Go 1.23.4 + CGO | Consume the same mechanism via the existing CGO bridge; new Go-side health-state guard | `sync/atomic`, consistent with existing `heartbeat.go` pattern |

## File Structure Plan

### Modified / New Files
```
src/include/xce/cluster.h                          # + declare xce_cluster_config_to_client_config()
src/lib/cluster.c                                    # + implement xce_cluster_config_to_client_config()
src/lib/testsuite/test_cluster.c                      # + unit tests for the new helper

src/worker/src/controller_connector.c                # rewrite: delete per-candidate loop in
                                                       #   xcecd_join_cluster/init_client_config;
                                                       #   single init+start+join using full
                                                       #   candidate list; register rejoin callback
src/worker/src/main.c                                 # fix: populate cc_ctx->cluster_cfg BEFORE
                                                       #   xcecd_start() (removes init-order bug)
src/worker/include/controller_connector.h             # + rejoin callback function declaration

src/scheduler/src/scheduler.c                         # rewrite: sched_init uses the new helper +
                                                       #   single init+start+join; delete inline
                                                       #   per-candidate loop; register rejoin callback
src/scheduler/src/main.c                              # no ordering change needed (already correct);
                                                       #   simplified call site only

src/frontend/apiserver/backend/clib.go                # rewrite RegisterToBackend: single connect
                                                       #   using the full candidate list via CGO calls
                                                       #   into xce_parse_peers/xce_cluster_config_validate/
                                                       #   xce_cluster_config_to_client_config; cgo
                                                       #   preamble gains a static C shim (mirrors the
                                                       #   existing xce_backend_handler/go_push_handler
                                                       #   pattern) that calls the reconnect.go export,
                                                       #   passed to xcecli_set_reconnect_handler; the
                                                       #   heartbeat tick callback explicitly re-invokes
                                                       #   the same rejoin path on recovery (see below)
src/frontend/apiserver/backend/health.go              # NEW: backend-degraded circuit-breaker state
src/frontend/apiserver/backend/reconnect.go           # NEW: exported Go rejoin logic (re-sends
                                                       #   xcecli_join for HostRole_apiserver, clears
                                                       #   health.go state); called from clib.go's C
                                                       #   shim AND directly from the heartbeat callback
src/frontend/apiserver/backend/heartbeat.go           # MODIFIED: tick callback detects a
                                                       #   degraded-to-healthy transition after
                                                       #   xcecli_heartbeat succeeds and explicitly
                                                       #   triggers reconnect.go's rejoin logic
src/frontend/apiserver/utils/config.go                # no schema change (Peers already exists)

src/worker/testsuite/test_e2e_multimaster_join.c       # NEW: candidate-list join + reconnect E2E
src/scheduler/testsuite/test_e2e_multimaster_join.c    # NEW: same, scheduler
src/frontend/apiserver/backend/health_test.go          # NEW: circuit-breaker unit tests
src/frontend/apiserver/backend/clib_test.go            # + candidate-list parity tests (existing file)
```

> `controller/` and `libxce`'s failover mechanism itself are not modified — only consumed.

## System Flows

### Startup connect (Req 1)

```mermaid
sequenceDiagram
    participant Daemon
    participant Convert as xce_cluster_config_to_client_config
    participant Ctx as xcecli_ctx (client.c)
    participant CandA as Candidate A (standby)
    participant CandB as Candidate B (primary)

    Daemon->>Convert: full candidate list (validated)
    Convert-->>Daemon: xce_client_config_t (nmasters=N)
    Daemon->>Ctx: xcecli_ctx_init + xcecli_start
    Ctx->>CandA: connect
    CandA-->>Ctx: connected, not primary
    Ctx->>Daemon: xcecli_join (role)
    Ctx->>CandA: join request
    CandA-->>Ctx: standby redirect
    Ctx->>Ctx: rotate to next candidate
    Ctx->>CandB: connect + join
    CandB-->>Ctx: joined as primary
    Ctx-->>Daemon: xcecli_join success
```

### Runtime failover (Req 2, Req 3)

```mermaid
sequenceDiagram
    participant Daemon
    participant Ctx as xcecli_ctx lock held
    participant OldPrimary as Old Primary dies
    participant NewPrimary as New Primary

    Daemon->>Ctx: xcecli_* call existing traffic
    Ctx->>OldPrimary: send
    OldPrimary--xCtx: connection lost
    Ctx->>Ctx: XCECLI_FAILOVER_CALL rotates and backs off
    Ctx->>NewPrimary: reconnect
    NewPrimary-->>Ctx: connected
    Ctx->>Daemon: invoke rejoin callback lock still held
    Note over Daemon: callback only sets a needs-rejoin flag, no I/O, returns 0
    Ctx->>NewPrimary: resume original call
    NewPrimary-->>Ctx: reply
    Ctx-->>Daemon: original call result, lock released
    Note over Daemon: after the call returns, lock is free
    Daemon->>Daemon: check and clear needs-rejoin flag
    Daemon->>Ctx: xcecli_join role separate call
    Ctx->>NewPrimary: join
    NewPrimary-->>Ctx: joined
    Ctx-->>Daemon: rejoin complete
```

Key decision reflected in the diagram: the rejoin callback invoked **by the library** never performs I/O — it only flags that a rejoin is needed (see "Rejoin callback constraint" above, avoiding a self-deadlock on the still-held per-ctx lock). The actual `xcecli_join` call happens as a distinct follow-up call from the daemon, on the same thread (worker: connector thread; scheduler: allocator thread; apiserver: the requesting goroutine), once the triggering call has returned and released the lock — no separate reconnect loop or new thread is introduced either way.

### apiserver fail-fast while degraded (Req 4)

```mermaid
sequenceDiagram
    participant Client as API Client
    participant Handler as apiserver handler
    participant Health as health.go
    participant Ctx as xcecli_ctx (gCtx)

    Client->>Handler: HTTP request needing backend
    Handler->>Health: is backend degraded?
    alt degraded
        Health-->>Handler: yes
        Handler-->>Client: 503 service unavailable (fast)
    else healthy
        Health-->>Handler: no
        Handler->>Ctx: backend call
        alt call exhausts retry budget
            Ctx-->>Handler: error
            Handler->>Health: mark degraded (with TTL)
            Handler-->>Client: 503 service unavailable
        else call succeeds
            Ctx-->>Handler: result
            Handler-->>Client: 200 with result
        end
    end
```

### apiserver idle-period reconnect (Req 2.4)

```mermaid
sequenceDiagram
    participant HB as heartbeat.go tick
    participant Ctx as xcecli_ctx (gCtx, FF_CALL)
    participant Rejoin as reconnect.go
    participant Health as health.go
    participant NewPrimary as New Primary

    Note over HB: no client requests in flight
    HB->>Ctx: xcecli_heartbeat (fire-and-forget)
    Ctx->>Ctx: connection known-dead: best-effort rotate + reconnect
    Ctx->>NewPrimary: reconnected transport only
    Ctx-->>HB: rc success (was failing, now succeeds)
    HB->>HB: detect failure-to-success transition
    HB->>Rejoin: explicitly trigger rejoin
    Rejoin->>NewPrimary: xcecli_join(HostRole_apiserver)
    NewPrimary-->>Rejoin: joined
    Rejoin->>Health: MarkHealthy()
```

This path exists because `XCECLI_FF_CALL` (fire-and-forget, used by the heartbeat) never invokes the library's own rejoin hook — only the full-retry path (`XCECLI_FAILOVER_CALL`, used by write calls) does. Without this explicit trigger, an idle apiserver instance would repair its TCP connection on every heartbeat tick but never re-establish cluster membership, so push notifications would stay dead until an unrelated write request happened to arrive.

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1-1.3 | Try each candidate in order at startup | libxce (unchanged), ConfigConverter | `xcecli_start`, `xcecli_join` | Startup connect |
| 1.4 | Worker uses full list on FIRST attempt | WorkerControllerConnector | `controller_connector.c` init order fix | Startup connect |
| 1.5 | Single/no-candidate mode unchanged | All three | ConfigConverter (`nmasters=1` path) | — |
| 1.6 | Clear startup failure if all candidates fail | All three | existing `xcecli_start`/`xcecli_join` return codes | Startup connect |
| 2.1-2.3 | Automatic reconnect + retry until success or shutdown | libxce (unchanged), all three's rejoin callbacks | `xcecli_set_reconnect_handler` | Runtime failover |
| 2.4 | apiserver resumes push notifications after reconnect (incl. idle-period recovery) | ApiserverBackendBridge | `reconnect.go`'s actual-rejoin function re-sends `xcecli_join(HostRole_apiserver)`, triggered via the shared post-call helper's flag check on write-path reconnect AND explicitly from `heartbeat.go` on idle-period recovery | Runtime failover, apiserver idle-period reconnect |
| 3.1-3.2 | Retry/backoff doesn't block other liveness work | WorkerControllerConnector (thread isolation preserved), SchedulerControllerClient (allocator/picker thread separation) | existing thread structure | Runtime failover |
| 4.1-4.2 | apiserver fails fast to clients while degraded | ApiserverBackendHealth | `health.go` | apiserver fail-fast flow |
| 5.1-5.2 | Per-candidate connect/reconnect logging | All three | log statements at each rotation | Startup connect, Runtime failover |
| 6.1-6.2 | Consistent candidate-list syntax | ApiserverConfigParity | CGO call to `xce_parse_peers`/`xce_cluster_config_validate` | Startup connect |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|--------------|--------|---------------|-------------------|-----------|
| ConfigConverter | libxce (C) | Convert validated candidate list to client config | 1.1-1.6, 6.1 | `xce_cluster_config_validate` (P0) | Service |
| WorkerControllerConnector | worker (C) | Single init/connect/join + rejoin for worker | 1.1-1.4, 2.1-2.3, 3.1-3.2, 5.1-5.2 | ConfigConverter (P0), `xcecli_*` (P0) | Service |
| SchedulerControllerClient | scheduler (C) | Single init/connect/join + rejoin for scheduler | 1.1-1.3, 1.5-1.6, 2.1-2.3, 3.1-3.2, 5.1-5.2 | ConfigConverter (P0), `xcecli_*` (P0) | Service |
| ApiserverBackendBridge | apiserver (Go/CGO) | Single init/connect/join for apiserver; config parity | 1.1-1.6, 2.4, 5.1-5.2, 6.1-6.2 | ConfigConverter via CGO (P0), `xcecli_*` via CGO (P0) | Service |
| ApiserverBackendHealth | apiserver (Go) | Fail-fast circuit breaker for backend-down state | 4.1-4.2 | ApiserverBackendBridge (P0, consumes rejoin/error signals) | State |

### Shared Library

#### ConfigConverter

| Field | Detail |
|-------|--------|
| Intent | Turn a validated `xce_cluster_config_t` into a fully-populated `xce_client_config_t`, clamped to `N_MAX_MASTERS` with a WARN on truncation |
| Requirements | 1.1, 1.2, 1.3, 1.5, 1.6, 6.1 |

**Responsibilities & Constraints**
- Pure, socket-free transformation (mirrors `xce_test`'s existing inline logic at `test.c:360-372`, generalized into one reusable function).
- Does not call `xce_cluster_config_validate()` itself — callers validate first (existing contract, unchanged) and pass a pre-validated config.
- Does not own `from`/`key` (identity/auth) — caller sets those on the output struct after conversion, matching how `xce_test`, worker, and scheduler already source them independently (CLI/env-specific).

**Dependencies**
- Inbound: WorkerControllerConnector, SchedulerControllerClient, ApiserverBackendBridge (all P0)
- Outbound: none (leaf function)

**Contracts**: Service [x]

##### Service Interface
```c
/* xce/cluster.h */
/**
 * xce_cluster_config_to_client_config() — convert a validated candidate list
 * into an xce_client_config_t's master fields (nmasters/masters/master_ports).
 *
 * Copies up to N_MAX_MASTERS candidates from cfg->candidates[] in order.
 * If cfg->candidates_count > N_MAX_MASTERS, copies the first N_MAX_MASTERS and
 * logs a WARN identifying the truncated count (mirrors xce_test's existing
 * clamp behavior).
 *
 * Does NOT set out->from / out->key — caller sets those separately.
 * Does NOT validate cfg — caller must have already called
 * xce_cluster_config_validate() successfully.
 *
 * Returns 0 on success, -EINVAL if cfg or out is NULL or cfg->candidates_count < 1.
 */
int xce_cluster_config_to_client_config(const xce_cluster_config_t* cfg,
                                        xce_client_config_t* out);
```
- Preconditions: `cfg` already passed `xce_cluster_config_validate()`.
- Postconditions: `out->nmasters` in `[1, N_MAX_MASTERS]`; `out->masters[i]`/`out->master_ports[i]` populated for `i < out->nmasters`, in the same order as `cfg->candidates[]`.
- Invariants: never writes beyond `N_MAX_MASTERS` entries regardless of `cfg->candidates_count`.

### Worker

#### WorkerControllerConnector

| Field | Detail |
|-------|--------|
| Intent | Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect |
| Requirements | 1.1-1.4, 2.1-2.3, 3.1-3.2, 5.1-5.2 |

**Responsibilities & Constraints**
- Replaces `xcecd_join_cluster`'s per-candidate loop (`controller_connector.c:237-311`) with: build `xce_client_config_t` via ConfigConverter once (candidates from `cc_ctx->cluster_cfg` when `--peers` given, else today's single-host fallback path unchanged per Req 1.5), then one `xcecli_ctx_init` + `xcecli_start` + `xcecli_join`.
- Fixes the init-order bug by moving `cc_ctx->cluster_cfg` population (`main.c`) to before `xcecd_start()` is called, so the very first connect already sees the full candidate list (Req 1.4).
- Registers a rejoin callback (`xcecli_set_reconnect_handler`) that only sets a `needs_rejoin` flag and returns `0` — per "Rejoin callback constraint" above, it must not call `xcecli_join` (or any `xcecli_*` function) directly, since the library invokes it while `ac->failover.lock` is still held.
- `checking_connection()`'s periodic `xcecli_hello` keeps its existing role (liveness probe on the connector thread) but no longer needs to manually call `xcecd_join_cluster` on failure — a transport failure inside `xcecli_hello` already triggers the library's transparent reconnect + the registered flag-setting callback. Immediately after each `xcecli_hello` call returns (lock now free), `checking_connection()` checks and clears `needs_rejoin`; if set, it calls `xcecli_join(ctx, HostRole_worker, &host_attrs)` as a separate call — the same join payload used at startup.
- Runs entirely on the existing dedicated connector thread (`start_controller_connector_thread`/`connector_to_controller`), preserving isolation from `hardware_monitor`/`system_monitor`/`process_manager` update paths (Req 3.1) — no new thread introduced.

**Dependencies**
- Inbound: `xcecd_start()` (P0)
- Outbound: ConfigConverter (P0), `xcecli_*` (P0)

**Contracts**: Service [x]

**Implementation Notes**
- Integration: preserve existing WARN-level log statements around standby-redirect and candidate-exhaustion, now emitted from the library's classification path or immediately around the single init/join call, so Req 5 diagnostics are not lost in the simplification.
- Validation: implementation must confirm (code read, not test-only) that `hardware_monitor_update`/`system_monitor_update`/`process_manager_update` do not execute on the connector thread — flagged as a Validation Hook, not re-verified in this design pass.
- Risks: deleting the hand-rolled loop must not silently drop the `MAX_RETRY_COUNT`/`RETRY_INTERVAL` bounded-retry-then-`handle_disconnect()` behavior in `checking_connection()` — that outer bound (Req 2.3's "keep retrying" vs. any existing bounded-give-up behavior) must be reconciled explicitly during implementation, not left ambiguous.

### Scheduler

#### SchedulerControllerClient

| Field | Detail |
|-------|--------|
| Intent | Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect |
| Requirements | 1.1-1.3, 1.5-1.6, 2.1-2.3, 3.1-3.2, 5.1-5.2 |

**Responsibilities & Constraints**
- Replaces `sched_init`'s inline per-candidate loop (`scheduler.c:462-495`) with ConfigConverter + one `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` call.
- Registers a rejoin callback that only sets a `needs_rejoin` flag and returns `0` (same constraint as worker — the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join` itself). After each `xcecli_jobdispatch` call returns, the allocator checks and clears the flag; if set, it calls `xcecli_join(ctx, HostRole_scheduler, NULL)` as a separate call before continuing.
- No new thread: `xcecli_jobdispatch` (the only backend call site, `scheduler.c:1289`) continues to run from the existing `sched_allocator` thread; blocking during a reconnect/backoff cycle is an accepted, benign degradation since the allocator has no useful work without a controller anyway, and `sched_job_picker` runs on its own separate thread unaffected (Req 3.1-3.2 satisfied by the existing thread split, not new code).

**Dependencies**
- Inbound: `sched_init()` (P0)
- Outbound: ConfigConverter (P0), `xcecli_*` (P0)

**Contracts**: Service [x]

**Implementation Notes**
- Integration: preserve the existing `WARN`/`INFO` log lines around candidate parsing (`main.c`) and add per-attempt/per-outcome logging around the single init/join call (today's loop logged each attempt; the library's own classification path should be checked for equivalent visibility, else add it locally).
- Risks: today's total absence of reconnect logic means this is additive behavior change (scheduler previously required a manual restart after primary loss) — call out explicitly in the task/rollout notes so operators aren't surprised by scheduler now retrying indefinitely instead of exiting.

### apiserver

#### ApiserverBackendBridge

| Field | Detail |
|-------|--------|
| Intent | Single connect/join at startup using the full candidate list via the C library (through CGO), replacing the Go-level per-peer loop; config-syntax parity with scheduler/worker |
| Requirements | 1.1-1.6, 2.4, 5.1-5.2, 6.1-6.2 |

**Responsibilities & Constraints**
- `RegisterToBackend()` (`clib.go`) is rewritten: build a C `xce_cluster_config_t` from `config.Peers` (or the single `Address`/`Port` fallback, Req 1.5) by calling `xce_parse_peers` via CGO (not the existing hand-written Go `parsePeer`), validate with `xce_cluster_config_validate`, convert with ConfigConverter, then call `xce_backend_init` **once** with the resulting `nmasters > 1` config — replacing the current `for _, peer := range config.Peers { tryConnect(...) }` Go loop entirely.
- `reconnect.go` holds two plain Go functions: (1) a `//export`-annotated flag-setter — registered as the library's rejoin callback, does no I/O, only sets a `needs_rejoin` atomic flag and returns `0` (per "Rejoin callback constraint" above: the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join`/any CGO backend function itself); (2) a plain Go function that performs the actual rejoin — calls `xcecli_join(gCtx.ctx, HostRole_apiserver, NULL)` and clears `ApiserverBackendHealth`'s degraded state on success.
- The flag-setter is **registered with the C library from `clib.go`'s own cgo preamble**, via a small static C shim (`xce_backend_rejoin_handler`, mirroring the existing `xce_backend_handler`/`go_push_handler` pattern already used for push dispatch) passed to `xcecli_set_reconnect_handler` — cgo symbol resolution is per-file (see `wire_dispatch_support.go`'s documented rationale for the identical `go_push_handler` wrapper), so the shim and the registration call must live in the same file as the `extern` declaration, not standalone in `reconnect.go`.
- Rather than duplicating the same pre-call/post-call logic across all four wrapper functions, the pre-call `IsDegraded()` fail-fast check (ApiserverBackendHealth, below) and the post-call `needs_rejoin` check-and-clear are both implemented once, in a single shared internal helper each of the four `clib.go` wrappers calls around its `C.xcecli_*` invocation — not copy-pasted per wrapper.
- **Idle-period reconnect gap (Req 2.4) fix**: `XCECLI_FF_CALL` — the macro backing `xcecli_heartbeat`, `heartbeat.go`'s existing periodic prober (`cmd/main.go:76`) — performs a best-effort transport reconnect on a known-dead connection but never invokes the library's rejoin hook (verified against `client_internal.h`/`client_failover.c`: `xcecli_invoke_rejoin` is called only from `xcecli_retry_next`, the full-retry path used by write calls, never from the fire-and-forget path). Left as-is, a controller failover occurring while apiserver has no in-flight write request would silently repair the TCP connection without ever re-joining, permanently starving push notifications until an unrelated write request happens to arrive. Fix: the heartbeat tick callback (`heartbeat.go`) tracks the previous tick's outcome; when a tick transitions from failure to success (transport just recovered), it explicitly calls `reconnect.go`'s rejoin function itself, then clears `ApiserverBackendHealth`. This reuses the same rejoin logic Task 4.3 already builds — no duplicate join-payload code, just an explicit trigger point the write-call path already gets for free from the library.

**Dependencies**
- Inbound: `cmd/main.go` (P0)
- Outbound: ConfigConverter via CGO (P0), `xcecli_*` via CGO (P0), ApiserverBackendHealth (P1, signals it on outcome)

**Contracts**: Service [x]

##### Service Interface
```go
// backend/clib.go
// RegisterToBackend initialises the client using the full configured
// candidate list (config.Peers, or the single Address/Port fallback) and
// registers the rejoin callback. Single call — no per-peer Go loop.
func RegisterToBackend(config *BackendConfig) error
```
- Preconditions: `StartNotificationListener` already called (unchanged from today).
- Postconditions: on success, `gCtx.ctx` is joined with `nmasters` reflecting every configured candidate (clamped per ConfigConverter); the rejoin callback is registered.
- Invariants: candidate-list parsing/validation errors surface the same way `xce_parse_peers`/`xce_cluster_config_validate` report them in C (Req 6.2) — not a separately-worded Go error.

**Implementation Notes**
- Integration: `parsePeer` (Go) and its tests are removed/retired in favor of the CGO-routed parser (Req 6.1).
- Integration: the rejoin C shim must be defined in `clib.go`'s cgo preamble (not `reconnect.go`) — see Responsibilities above; this was the design's one build-risk item, now resolved by construction rather than left as an implementation-time surprise.
- Risks: CGO error surfaces (C `-EINVAL` etc.) need a clear Go-side error message mapping so operators get an actionable message (Req 1.6, 6.2), not a bare error code.

#### ApiserverBackendHealth

| Field | Detail |
|-------|--------|
| Intent | Let request handlers fail fast (503) instead of blocking on the library's full retry budget when the backend is known-degraded |
| Requirements | 4.1, 4.2 |

**Responsibilities & Constraints**
- New `health.go`: an atomic "degraded-until" timestamp. `IsDegraded() bool` — true if `now < degraded-until`. `MarkDegraded(ttl time.Duration)` — called by backend-call wrapper functions in `clib.go` when a call's returned error indicates retry-budget exhaustion (e.g. `E_XCE_NOT_READY`). `MarkHealthy()` — called by the rejoin callback (`reconnect.go`) on a successful rejoin.
- Backend-call wrapper functions used by request-handling code paths (job submit, job control, queue control, host control, etc. — the existing `gCtx.ctx`-based calls in `clib.go`) check `IsDegraded()` first, via the shared internal helper described under ApiserverBackendBridge above (not duplicated per wrapper); if true, return a clear "backend unavailable" error immediately instead of invoking the blocking C call (Req 4.1). Calls that don't require the backend are unaffected (Req 4.2 — nothing to change there, they already don't touch `gCtx`).

**Dependencies**
- Inbound: ApiserverBackendBridge (sets state on outcomes)
- Outbound: none

**Contracts**: State [x]

##### State Management
- State model: single atomic degraded-until timestamp (int64 unix-nano via `sync/atomic`, same pattern as existing `heartbeatIntervalSecs` in `heartbeat.go`).
- Persistence & consistency: process-local, in-memory only; reset on apiserver restart (matches the rest of apiserver's in-memory state model per `tech.md`).
- Concurrency strategy: lock-free atomic read/write; safe for concurrent goroutines per request.

**Implementation Notes**
- Integration: TTL-based clearing (Risks & Mitigations in research.md) in addition to the rejoin-callback clear, so a missed/edge-case rejoin doesn't permanently wedge the flag.
- Validation: unit tests cover degraded→healthy transitions via both the TTL expiry and explicit `MarkHealthy()`.

## Error Handling

### Error Strategy
No new error taxonomy — reuse the existing controller status codes (`E_XCE_REDIRECT`/`E_XCE_STANDBY`/`E_XCE_NOT_READY`, `errnox.h`) that `client_failover.c` already classifies, and the existing `xce_cluster_config_validate()` `-EINVAL` cases for malformed candidate-list config.

### Error Categories and Responses
- **Malformed candidate-list config** (Req 1.6, 6.2): `xce_parse_peers`/`xce_cluster_config_validate` failure at startup → each component reports a clear message naming the offending entry and refuses to start (unchanged behavior for worker/scheduler; apiserver gains the same via the CGO route).
- **All candidates unreachable at startup** (Req 1.6): `xcecli_start`/`xcecli_join` returns the last transport error after exhausting all candidates → component reports which candidates were attempted (Req 5.2) and fails startup (worker/scheduler: process exits as today; apiserver: `RegisterToBackend` returns an error, `cmd/main.go`'s existing handling applies unchanged).
- **Runtime disconnect** (Req 2): transparently retried/rotated by the library; components only observe it through their own periodic calls' return codes and the rejoin callback — no new error category, just new logging around the existing outcomes.
- **Candidate count exceeds `N_MAX_MASTERS`**: ConfigConverter clamps and WARNs (not an error) — matches `xce_test`'s existing behavior exactly.
- **apiserver backend degraded** (Req 4.1): mapped to a clear 503-style API error, distinct from other 5xx causes, so operators/clients can distinguish "backend down, retrying" from an unrelated internal error.

### Monitoring
Per-candidate connect/reconnect attempts and outcomes are logged (Req 5) using each component's existing logger (`XCECD_LOG` for worker, the scheduler's existing `INFO`/`WARN` macros, `zap` for apiserver) — no new logging infrastructure, just new call sites around the simplified connect/reconnect path.

## Testing Strategy

- **Unit Tests**:
  - `ConfigConverter`: clamps at `N_MAX_MASTERS` with WARN when given more candidates; preserves candidate order; rejects `NULL`/empty input (`test_cluster.c`).
  - apiserver `health.go`: degraded→healthy via `MarkHealthy()`, via TTL expiry; concurrent goroutine safety (`health_test.go`).
  - apiserver CGO-routed parser: same malformed-input cases scheduler/worker already reject, now asserted through the Go call site (`clib_test.go`).
  - Rejoin flag-setter callbacks (worker, scheduler, apiserver): with a real `ac->failover.lock` held (simulating the library's call context per "Rejoin callback constraint"), invoking the registered callback returns promptly without blocking and performs no I/O — a regression guard against reintroducing the self-deadlock this design pass found.
- **Integration Tests**:
  - Worker/scheduler startup against a 3-candidate config where candidate[0] is a live standby — assert the component joins as expected without operator intervention (Req 1.1-1.3), extending each component's `testsuite/`.
  - Worker/scheduler reconnect: kill the currently-joined primary mid-session, assert the component reconnects and re-joins the new primary without restart (Req 2.1-2.3), reusing the project's existing `/xce-deploy-ha` + `/xce-test-ha` pattern already used for controller HA soak testing — this is also the test that would hang (not fail cleanly) if the self-deadlock were reintroduced, so it must run with a wall-clock timeout that fails loud rather than hanging CI.
- **E2E Tests**:
  - Full 3-node HA cluster with worker + scheduler + apiserver all configured with the full candidate list; kill the primary; assert all three reconnect, worker resumes job execution reporting, scheduler resumes dispatch, apiserver resumes serving requests and receiving push notifications (Req 2.4) — new `test_e2e_multimaster_join.c` per C component plus an apiserver-side check in the existing HA soak harness.
  - apiserver idle-period reconnect: kill the primary while apiserver has no in-flight write request (only the heartbeat prober running); assert apiserver still rejoins and resumes push notifications, proving the heartbeat-triggered rejoin path (not just the write-call-triggered one) works (Req 2.4).
  - apiserver fail-fast: with the backend killed and not yet reconnected, assert an API request returns quickly (bounded, sub-second) with a clear 503 rather than hanging for the full retry budget (Req 4.1).

## Security Considerations
No new trust boundary: candidate hosts are the same set of controllers already trusted today; TLS/identity handling is unchanged (see Non-Goals — CA-cert wiring is out of scope and pre-existing). No new authentication surface is introduced.