myxce/.kiro/specs/component-failover-adoption/design.md

# Design Document: component-failover-adoption

## Overview

Scheduler, worker, and apiserver each connect to the controller cluster today using only a single configured endpoint at a time, with divergent, partly-broken hand-rolled logic for trying alternates. Meanwhile `libxce` already ships a complete, tested, transparent multi-master mechanism (`XCECLI_FAILOVER_CALL` and friends in `src/lib/client.c`) — the same one `xce_test`'s `masters` setting already uses correctly. This design wires all three components onto that existing mechanism: populate the full candidate list once, let the library rotate/retry/reconnect on every call, and register a rejoin callback so a component re-establishes cluster membership after a transparent reconnect.

**Users**: HA cluster operators who configure multiple controller candidates (`--peers` for scheduler/worker, `backend.peers` YAML for apiserver) and expect scheduler, worker, and apiserver to survive a controller failover without manual restarts — the same expectation `xce_test` already meets.

**Impact**: Removes worker's and scheduler's hand-rolled per-candidate join loops (deleting, not adding, most of the connection-retry code in `controller_connector.c` and `scheduler.c`); replaces apiserver's Go-level per-peer loop with a single library-driven connect; adds one new small shared C helper and a small Go-side health guard. No changes to the controller or to `libxce`'s failover mechanism itself.

### Goals
- Scheduler, worker, and apiserver each hand the full configured candidate list to `xce_client_config_t` on the very first connection attempt (Req 1).
- All three automatically reconnect and resume cluster membership after their controller connection is lost, without a process restart (Req 2).
- Worker's and scheduler's retry/backoff never blocks duties independent of controller connectivity (Req 3).
- apiserver returns a clear, fast error to its own API clients while its backend connection is down, instead of hanging (Req 4).
- Every connect/reconnect attempt and outcome is logged per-candidate (Req 5).
- apiserver accepts the identical candidate-list syntax scheduler/worker already accept (Req 6).
- Scheduler decision publishing (`xcecli_jobdispatch`/`xcecli_set_pend_reason`) never silently drops a decision on retry-budget exhaustion — it keeps retrying that decision until success or shutdown, so scheduler-local and controller-observed job state cannot permanently diverge (Req 7).

### Non-Goals
- Changing `xce_test` (it is already the reference-correct client) — its clamp already references `N_MAX_MASTERS` symbolically (`test.c:372-378`), so it auto-absorbs the constant reconciliation below with zero code change.
- Completing or adopting the separate, unused `xce_connect_cluster()`/`xce_reconnect_*` primitive in `cluster.c` — superseded by the mechanism this design adopts; left untouched.
- Wiring CA-cert/TLS trust into the real connect path — `xce_client_config_t` carries no such field today and this feature does not add one; TLS is established the same way it already is.
- Controller-side changes (redirect-reply population, peer authentication) — already shipped by `libxce-failover`.
- Recovering worker job-state lost during a disconnect window — owned by `worker-rejoin-resync`.
- Persisting queued-but-unpublished scheduler decisions across a scheduler process crash/restart (Req 7 AC6) — Req 7's fix keeps a running scheduler process from giving up early on a network retry; it adds no WAL/persistence for the in-memory `decisions`/`jbuf` queues, so a scheduler crash mid-outage still loses whatever was queued, exactly as it does today.

## Boundary Commitments

### This Spec Owns
- A new shared libxce helper that converts a validated `xce_cluster_config_t` (parsed candidate list) into a fully-populated `xce_client_config_t`.
- Worker's, scheduler's, and apiserver's integration code that calls `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` with the full candidate list and registers a rejoin callback.
- apiserver's candidate-list config parsing (routed through the existing C parser via CGO instead of a second Go implementation).
- apiserver's client-facing fail-fast behavior while its backend connection is degraded.
- Connect/reconnect diagnostic logging in all three components.
- Scheduler's decision-publish retry-until-success-or-shutdown wrapper around `xcecli_jobdispatch`/`xcecli_set_pend_reason` (Req 7) — closes the silent-drop gap found in the 2026-08-10 design validation.
- **Reconciling `N_MAX_MASTERS` (`client.h`) to derive from `MAX_HA_CANDIDATES` (`config.h`)** (Req 1 AC7, scope expanded 2026-08-10 at explicit user direction — supersedes this design's earlier Non-Goal treating the mismatch as pre-existing/out-of-scope): `config.h` becomes the single source of truth for "max configured controller candidates." Confirmed safe by source audit — no wire-protocol array assumes 3 (`masters[]` is never serialized; the controller's own peer list is a dynamic capnp `List(CtrlPeer)`), and `config.h` has no include-cycle risk (it depends only on `<stdint.h>`; `client.h` already transitively sees `MAX_HA_CANDIDATES` via `xceapi.h`/`xce/types.h` → `config.h`).
- **Eliminating the legacy single-host code path in worker, scheduler, and apiserver in favor of a single unified pipeline** (Req 1 AC5, redefined 2026-08-10): today all three synthesize a `nmasters=1` `xce_client_config_t` directly from legacy fields, bypassing `xce_cluster_config_t`/`xce_parse_peers`/`xce_cluster_config_validate`/ConfigConverter entirely (`controller_connector.c:129-160`'s `init_client_config()`, `scheduler.c:613`'s single-host `else` branch, `clib.go:79-101`'s `tryConnect()` direct-CGO-struct path). This spec now closes that parallel path: a single controller is the `candidates_count=1` case of the exact same validated pipeline, not a separately-maintained mechanism.

### Out of Boundary
- `xce_test` and its `masters` setting (unchanged, reference implementation).
- `libxce`'s failover **mechanism** (`client.c`'s connect/rotate/retry/backoff logic, `client_internal.h`, `client_failover.c`) — consumed as-is, not modified. Distinct from `client.h`'s `N_MAX_MASTERS` **constant**, which this spec does change (above) — the mechanism and the struct-layout constant are different surfaces with different boundaries.
- `xce_connect_cluster()`/`xce_reconnect_*` (unused, untouched).
- Controller-side membership/redirect logic.
- Worker job-state resync content on rejoin (`worker-rejoin-resync`).
- TLS/CA-cert plumbing into the real connect path.

### Allowed Dependencies
- `src/lib/client.c` / `client_internal.h` / `client_failover.h` (shipped `libxce-failover` mechanism) — read-only dependency, no modification.
- `src/lib/cluster.c` / `xce/cluster.h` (`xce_parse_peers`, `xce_cluster_config_validate`) — existing, unmodified; this spec adds one new function alongside them.
- Existing per-component candidate-list CLI/YAML surfaces (`--peers`, `--ca-cert`, `backend.peers`) — unmodified externally; only how they're consumed changes.

### Revalidation Triggers
- Any change to `xce_client_config_t`'s layout, or any future change to `N_MAX_MASTERS`/`MAX_HA_CANDIDATES` that reintroduces divergence between them (this spec establishes them as one value — a regression that redefines `N_MAX_MASTERS` independently of `config.h` again should re-trigger review).
- Any change to `XCECLI_FAILOVER_CALL`/`XCECLI_FF_CALL` retry/classification semantics (`libxce-failover`).
- Any change to `xce_parse_peers`/`xce_cluster_config_validate` syntax or validation rules (apiserver's CGO-based parser would need re-verification).
- `worker-rejoin-resync` landing — worker's rejoin callback becomes the natural place to also trigger job-state resync; this spec's callback is a no-op-beyond-rejoin placeholder for that.

## Architecture

### Existing Architecture Analysis
Today each of the three components independently drives its own `xcecli_ctx_t` lifecycle with `nmasters` hardcoded to `1` per connection attempt, and layers its own outer candidate loop on top:
- **worker**: `controller_connector.c`'s `xcecd_join_cluster` loops candidates itself, but the very first connection (inside `xcecd_start()`) happens before `cc_ctx->cluster_cfg` is even populated (`main.c` assigns it after `xcecd_start()` returns) — the candidate list is silently ignored on startup.
- **scheduler**: `scheduler.c`'s `sched_init` loops candidates correctly at startup, but there is no reconnect logic anywhere afterward.
- **apiserver**: `clib.go`'s `RegisterToBackend` loops candidates once via a hand-written Go parser, only at process startup.

None of the three ever set `nmasters > 1` on a single `xce_client_config_t`, so the library's own rotation/retry (`client.c`, wrapped by `XCECLI_FAILOVER_CALL`) never actually engages for them.

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    subgraph Daemons
        Worker[worker xcecd]
        Scheduler[xce_scheduler]
        Apiserver[apiserver backend package]
    end

    subgraph NewSharedCode
        Convert[xce_cluster_config_to_client_config]
    end

    subgraph ExistingLibxce
        Parse[xce_parse_peers]
        Validate[xce_cluster_config_validate]
        CtxInit[xcecli_ctx_init xcecli_start xcecli_join]
        Failover[XCECLI_FAILOVER_CALL rotate retry backoff]
        Rejoin[rejoin callback xcecli_set_reconnect_handler]
    end

    subgraph Cluster
        Primary[Controller Primary]
        Standby1[Controller Standby]
    end

    Worker --> Parse
    Scheduler --> Parse
    Apiserver --> Parse
    Parse --> Validate
    Validate --> Convert
    Convert --> CtxInit
    CtxInit --> Failover
    Failover --> Rejoin
    Failover --> Primary
    Failover --> Standby1
```

**Architecture Integration**:
- Selected pattern: thin per-component integration layer over an unchanged, already-shipped library mechanism (adapter, not new subsystem).
- Domain boundaries: `libxce` owns candidate parsing, validation, connect/rotate/retry/backoff, and rejoin-hook invocation (unchanged). Each daemon owns only: building the candidate list from its own config surface, calling the library once, and supplying its role-specific rejoin behavior (what to send when re-joining).
- Existing patterns preserved: the `xce_parse_peers`/`xce_cluster_config_validate`/`xce_client_config_t` split; the CGO bridge pattern already used throughout `backend/`; worker's dedicated connector thread.
- New components: one C helper (candidate-list → client-config conversion, generalized out of `xce_test`'s inline logic) and one small Go health-state module (apiserver-only, has no C-side equivalent need).
- Steering compliance: no new blocking work introduced on iomgr/delaymgr-equivalent threads; worker's existing thread isolation is preserved, not altered.

### Rejoin callback constraint: avoiding self-deadlock on `ac->failover.lock`

Verified directly against `client_failover.c:341-411`: `XCECLI_FAILOVER_CALL` holds `ac->failover.lock` (a plain, non-recursive `pthread_mutex_t` — `mylock_init` calls `pthread_mutex_init(l, NULL)`, no `PTHREAD_MUTEX_RECURSIVE` attribute) for the **entire** retry loop, and `xcecli_invoke_rejoin()` — which calls the registered rejoin callback — is invoked **synchronously from inside that loop, while the lock is held** (`client_failover.c:397`). Any rejoin callback that itself calls a public `xcecli_*` wrapper on the *same* `xcecli_ctx_t` (e.g. `xcecli_join`, which is itself `XCECLI_FAILOVER_CALL`-wrapped and tries to re-lock the same mutex) **self-deadlocks** — the calling thread blocks forever on its own already-held lock. This was never caught because `xcecli_set_reconnect_handler` has zero production callers today and the existing unit test's callback is a no-op probe that never calls back into the client.

This directly constrains every component's rejoin implementation in this spec:
- The callback **registered with the library** (worker's `checking_connection`→`xcecli_hello` path, scheduler's `xcecli_jobdispatch` path, apiserver's write-call path) must do **no I/O and call no `xcecli_*` function** — it may only set a lightweight "needs rejoin" flag (a plain bool/atomic on the consumer side) and return `0` so the interrupted operation resumes normally.
- Each component's own calling code, **after** the triggering `xcecli_*` call returns (at which point the library has released the lock), checks that flag and, if set, issues `xcecli_join(...)` **as a separate, subsequent call** — safe, since the lock is free by then. **The flag is cleared only once that `xcecli_join` call itself succeeds, never merely because it was attempted** (Req 2 AC5, gap found in the 2026-08-10 design validation's second pass): `xcecli_join` is itself `XCECLI_FAILOVER_CALL`-wrapped and can exhaust its own retry budget and return `-E_XCE_NOT_READY` just like any other wrapped call. Clearing the flag unconditionally before or regardless of that outcome would leave the component transport-reconnected but not cluster-joined, with nothing left to retry the join until the next full disconnect/reconnect cycle happens to fire the rejoin callback again — an indefinite, silent orphaning, not a bounded delay. On failure the flag stays set, so the component's next naturally-occurring opportunity (worker: next `xcecli_hello` return; scheduler: next decision-publish return, itself now retried indefinitely per Req 7; apiserver: next backend-call wrapper return) re-attempts the join.
- apiserver's heartbeat-triggered path (see below) is unaffected by this constraint — it already detects recovery and calls the rejoin logic **after** `xcecli_heartbeat` has returned (`XCECLI_FF_CALL` also releases its lock before returning), so it was never at risk.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|------------------|-------|
| Shared library | C (gnu17), `src/lib/cluster.c` | New candidate→client-config conversion helper | Extends existing file alongside `xce_parse_peers` |
| Worker / Scheduler | C (gnu17) | Consume the existing `xcecli_ctx_t` mechanism directly | No new dependency |
| apiserver | Go 1.23.4 + CGO | Consume the same mechanism via the existing CGO bridge; new Go-side health-state guard | `sync/atomic`, consistent with existing `heartbeat.go` pattern |

## File Structure Plan

### Modified / New Files
```
src/include/client.h                                  # fix: N_MAX_MASTERS derives from
                                                       #   config.h's MAX_HA_CANDIDATES (3->4)
                                                       #   instead of an independent literal (Req 1.7)
src/include/xce/cluster.h                          # + declare xce_cluster_config_to_client_config()
src/lib/cluster.c                                    # + implement xce_cluster_config_to_client_config()
src/lib/testsuite/test_cluster.c                      # + unit tests for the new helper
src/lib/testsuite/test_libxce_retry.c                 # fix: fo_make_client_masters() currently
                                                       #   hardcodes 3 literal host/port entries then
                                                       #   copies N_MAX_MASTERS slots -- update to match
                                                       #   the raised N_MAX_MASTERS, not silently copy
                                                       #   an uninitialized 4th slot

src/worker/src/controller_connector.c                # rewrite: delete init_client_config()'s
                                                       #   standalone xce_client_config_t construction
                                                       #   and xcecd_join_cluster()'s separate
                                                       #   single-host-fallback branch (both bypassed
                                                       #   xce_cluster_config_t entirely) -- single
                                                       #   init+start+join via the one synthesize-
                                                       #   validate-convert-connect sequence, whether
                                                       #   the candidate list came from --peers or was
                                                       #   synthesized as one candidate (Req 1.5);
                                                       #   register rejoin callback
src/worker/src/main.c                                 # fix: populate cc_ctx->cluster_cfg BEFORE
                                                       #   xcecd_start() (removes init-order bug);
                                                       #   fix hardcoded `if (candidates_count > 3)`
                                                       #   (main.c:414) to reference N_MAX_MASTERS
                                                       #   symbolically instead of a literal 3, so it
                                                       #   tracks the raised value automatically
src/worker/include/controller_connector.h             # + rejoin callback function declaration

src/scheduler/src/scheduler.c                         # rewrite: sched_init uses the new helper +
                                                       #   single init+start+join; delete inline
                                                       #   per-candidate loop AND the separate
                                                       #   "single-host path (original)" else branch
                                                       #   (scheduler.c:613, bypassed
                                                       #   xce_cluster_config_t entirely) -- single-host
                                                       #   is now candidates_count=1 through the same
                                                       #   sequence (Req 1.5); register rejoin callback;
                                                       #   sched_publish_dispatch/sched_publish_pend_reason
                                                       #   gain a retry-until-success-or-shutdown wrapper
                                                       #   around their xcecli_* call (Req 7)
src/scheduler/src/main.c                              # fix hardcoded `if (candidates_count > 3)`
                                                       #   (main.c:314) to reference N_MAX_MASTERS
                                                       #   symbolically, same as worker's fix above;
                                                       #   no ordering change needed (already correct)

src/frontend/apiserver/backend/clib.go                # rewrite RegisterToBackend: builds exactly ONE
                                                       #   xce_cluster_config_t regardless of source
                                                       #   (config.Peers parsed via xce_parse_peers, or
                                                       #   a synthesized single candidate from
                                                       #   config.Address/config.Port when Peers is
                                                       #   empty -- Req 1.5) then the same
                                                       #   validate/xce_cluster_config_to_client_config/
                                                       #   connect sequence either way; deletes
                                                       #   tryConnect()'s direct C.xce_client_config_t
                                                       #   construction and the Go-level per-peer loop
                                                       #   entirely; cgo preamble gains a static C shim
                                                       #   (mirrors the existing
                                                       #   xce_backend_handler/go_push_handler pattern)
                                                       #   that calls the reconnect.go export, passed to
                                                       #   xcecli_set_reconnect_handler
src/frontend/apiserver/backend/peers.go               # DELETED: hand-written parsePeer() retired in
                                                       #   favor of the CGO-routed xce_parse_peers (Req 6.1)
src/frontend/apiserver/backend/health.go              # NEW: backend-degraded circuit-breaker state
src/frontend/apiserver/backend/reconnect.go           # NEW: exported Go rejoin logic (re-sends
                                                       #   xcecli_join for HostRole_apiserver, clears
                                                       #   health.go state); called from clib.go's C
                                                       #   shim AND directly from the heartbeat callback
src/frontend/apiserver/backend/heartbeat.go           # MODIFIED: tick callback sets a needs_rejoin
                                                       #   flag on a failure->success transport
                                                       #   transition (FF_CALL never sets it itself),
                                                       #   then on every subsequent tick where
                                                       #   xcecli_heartbeat succeeds AND needs_rejoin is
                                                       #   still true (not gated on
                                                       #   ApiserverBackendHealth.IsDegraded(), which has
                                                       #   an independent TTL -- Req 2.5) retries
                                                       #   reconnect.go's rejoin function until it
                                                       #   actually succeeds
src/frontend/apiserver/utils/config.go                # no schema change (Peers/Address/Port already exist)

src/worker/testsuite/test_e2e_multimaster_join.c       # NEW: candidate-list join + reconnect E2E;
                                                       #   also covers the unified single-host path
                                                       #   (no --peers) hitting the same code (Req 1.5)
src/scheduler/testsuite/test_e2e_multimaster_join.c    # NEW: same, scheduler
src/frontend/apiserver/backend/health_test.go          # NEW: circuit-breaker unit tests
src/frontend/apiserver/backend/clib_test.go            # + candidate-list parity tests (existing file)
```

> `controller/` and `libxce`'s failover mechanism itself are not modified — only consumed.

## System Flows

### Startup connect (Req 1)

```mermaid
sequenceDiagram
    participant Daemon
    participant Convert as xce_cluster_config_to_client_config
    participant Ctx as xcecli_ctx (client.c)
    participant CandA as Candidate A (standby)
    participant CandB as Candidate B (primary)

    Daemon->>Convert: full candidate list (validated)
    Convert-->>Daemon: xce_client_config_t (nmasters=N)
    Daemon->>Ctx: xcecli_ctx_init + xcecli_start
    Ctx->>CandA: connect
    CandA-->>Ctx: connected, not primary
    Ctx->>Daemon: xcecli_join (role)
    Ctx->>CandA: join request
    CandA-->>Ctx: standby redirect
    Ctx->>Ctx: rotate to next candidate
    Ctx->>CandB: connect + join
    CandB-->>Ctx: joined as primary
    Ctx-->>Daemon: xcecli_join success
```

### Runtime failover (Req 2, Req 3)

```mermaid
sequenceDiagram
    participant Daemon
    participant Ctx as xcecli_ctx lock held
    participant OldPrimary as Old Primary dies
    participant NewPrimary as New Primary

    Daemon->>Ctx: xcecli_* call existing traffic
    Ctx->>OldPrimary: send
    OldPrimary--xCtx: connection lost
    Ctx->>Ctx: XCECLI_FAILOVER_CALL rotates and backs off
    Ctx->>NewPrimary: reconnect
    NewPrimary-->>Ctx: connected
    Ctx->>Daemon: invoke rejoin callback lock still held
    Note over Daemon: callback only sets a needs-rejoin flag, no I/O, returns 0
    Ctx->>NewPrimary: resume original call
    NewPrimary-->>Ctx: reply
    Ctx-->>Daemon: original call result, lock released
    Note over Daemon: after the call returns, lock is free
    Daemon->>Daemon: check and clear needs-rejoin flag
    Daemon->>Ctx: xcecli_join role separate call
    Ctx->>NewPrimary: join
    NewPrimary-->>Ctx: joined
    Ctx-->>Daemon: rejoin complete
```

Key decision reflected in the diagram: the rejoin callback invoked **by the library** never performs I/O — it only flags that a rejoin is needed (see "Rejoin callback constraint" above, avoiding a self-deadlock on the still-held per-ctx lock). The actual `xcecli_join` call happens as a distinct follow-up call from the daemon, on the same thread (worker: connector thread; scheduler: the main driver-loop thread, inside `sched_publish_dispatch` immediately after `xcecli_jobdispatch` returns — **not** the allocator thread, see SchedulerControllerClient below; apiserver: the requesting goroutine), once the triggering call has returned and released the lock — no separate reconnect loop or new thread is introduced either way.

### apiserver fail-fast while degraded (Req 4)

```mermaid
sequenceDiagram
    participant Client as API Client
    participant Handler as apiserver handler
    participant Health as health.go
    participant Ctx as xcecli_ctx (gCtx)

    Client->>Handler: HTTP request needing backend
    Handler->>Health: is backend degraded?
    alt degraded
        Health-->>Handler: yes
        Handler-->>Client: 503 service unavailable (fast)
    else healthy
        Health-->>Handler: no
        Handler->>Ctx: backend call
        alt call exhausts retry budget
            Ctx-->>Handler: error
            Handler->>Health: mark degraded (with TTL)
            Handler-->>Client: 503 service unavailable
        else call succeeds
            Ctx-->>Handler: result
            Handler-->>Client: 200 with result
        end
    end
```

### apiserver idle-period reconnect (Req 2.4)

```mermaid
sequenceDiagram
    participant HB as heartbeat.go tick
    participant Ctx as xcecli_ctx (gCtx, FF_CALL)
    participant Rejoin as reconnect.go
    participant Health as health.go
    participant NewPrimary as New Primary

    Note over HB: no client requests in flight
    HB->>Ctx: xcecli_heartbeat (fire-and-forget)
    Ctx->>Ctx: connection known-dead: best-effort rotate + reconnect
    Ctx->>NewPrimary: reconnected transport only
    Ctx-->>HB: rc success (transport reconnected)
    Note over HB: was failing, now succeeds -> set needs_rejoin=true (FF_CALL never sets it for us)
    HB->>HB: needs_rejoin still true?
    HB->>Rejoin: trigger rejoin
    Rejoin->>NewPrimary: xcecli_join(HostRole_apiserver)
    alt join succeeds
        NewPrimary-->>Rejoin: joined
        Rejoin->>HB: clear needs_rejoin
        Rejoin->>Health: MarkHealthy()
    else join's own retry budget exhausted
        NewPrimary--xRejoin: -E_XCE_NOT_READY
        Note over HB: needs_rejoin stays true (independent of Health's TTL-governed state) — next tick retries
    end
```

This path exists because `XCECLI_FF_CALL` (fire-and-forget, used by the heartbeat) never invokes the library's own rejoin hook — only the full-retry path (`XCECLI_FAILOVER_CALL`, used by write calls) does. Without this explicit trigger, an idle apiserver instance would repair its TCP connection on every heartbeat tick but never re-establish cluster membership, so push notifications would stay dead until an unrelated write request happened to arrive.

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1-1.3 | Try each candidate in order at startup | libxce (unchanged), ConfigConverter | `xcecli_start`, `xcecli_join` | Startup connect |
| 1.4 | Worker uses full list on FIRST attempt | WorkerControllerConnector | `controller_connector.c` init order fix | Startup connect |
| 1.5 | Single controller is `candidates_count=1` of the one mechanism, not a separate fallback (redefined 2026-08-10, was "unchanged single-host mode") | All three | ConfigConverter (`nmasters=1` case, no branch) — worker's `init_client_config()`/single-host branch, scheduler's single-host `else` branch, and apiserver's `tryConnect()` all deleted | Startup connect |
| 1.6 | Clear startup failure if all candidates fail | All three | existing `xcecli_start`/`xcecli_join` return codes | Startup connect |
| 1.7 | `MAX_HA_CANDIDATES` (`config.h`) is the single source of truth for max candidates; `N_MAX_MASTERS` derives from it | ConfigConverter (clamp becomes defensive, not normally triggered) | `client.h`'s `N_MAX_MASTERS` `#define` | — |
| 2.1-2.3 | Automatic reconnect + retry until success or shutdown | libxce (unchanged), all three's rejoin callbacks | `xcecli_set_reconnect_handler` | Runtime failover |
| 2.4 | apiserver resumes push notifications after reconnect (incl. idle-period recovery) | ApiserverBackendBridge | `reconnect.go`'s actual-rejoin function re-sends `xcecli_join(HostRole_apiserver)`, triggered via the shared post-call helper's flag check on write-path reconnect AND explicitly from `heartbeat.go` on idle-period recovery | Runtime failover, apiserver idle-period reconnect |
| 2.5 | Re-join call itself retries on failure, not treated as one-shot | WorkerControllerConnector, SchedulerControllerClient, ApiserverBackendBridge | flag/degraded-state clears only on `xcecli_join` success, not on mere attempt; apiserver's idle-path checks "still degraded" every tick, not just the failure→success edge | Runtime failover, apiserver idle-period reconnect |
| 3.1-3.3 | Retry/backoff doesn't block other liveness work | WorkerControllerConnector (thread isolation, full isolation per AC1), SchedulerControllerClient (bounded, occupancy-dependent buffering via `jbuf`/`decisions` cbufs per AC2 — narrower than worker's guarantee by design, no job lost per Req 7, resolved 2026-08-10) | existing thread structure | Runtime failover |
| 4.1-4.2 | apiserver fails fast to clients while degraded | ApiserverBackendHealth | `health.go` | apiserver fail-fast flow |
| 5.1-5.2 | Per-candidate connect/reconnect logging | All three | log statements at each rotation | Startup connect, Runtime failover |
| 6.1-6.2 | Consistent candidate-list syntax | ApiserverConfigParity | CGO call to `xce_parse_peers`/`xce_cluster_config_validate` | Startup connect |
| 7.1-7.6 | Scheduler decision publish survives retry-budget exhaustion without loss | SchedulerControllerClient | `sched_publish_dispatch`/`sched_publish_pend_reason` retry-until-success-or-shutdown wrapper | Runtime failover |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|--------------|--------|---------------|-------------------|-----------|
| ConfigConverter | libxce (C) | Convert validated candidate list to client config | 1.1-1.7, 6.1 | `xce_cluster_config_validate` (P0) | Service |
| WorkerControllerConnector | worker (C) | Single init/connect/join + rejoin for worker | 1.1-1.5, 2.1-2.3, 2.5, 3.1, 3.3, 5.1-5.2 | ConfigConverter (P0), `xcecli_*` (P0) | Service |
| SchedulerControllerClient | scheduler (C) | Single init/connect/join + rejoin for scheduler; decision-publish never silently drops on retry-budget exhaustion | 1.1-1.3, 1.5-1.7, 2.1-2.3, 2.5, 3.2-3.3, 5.1-5.2, 7.1-7.6 | ConfigConverter (P0), `xcecli_*` (P0) | Service |
| ApiserverBackendBridge | apiserver (Go/CGO) | Single init/connect/join for apiserver; config parity | 1.1-1.7, 2.4-2.5, 5.1-5.2, 6.1-6.2 | ConfigConverter via CGO (P0), `xcecli_*` via CGO (P0) | Service |
| ApiserverBackendHealth | apiserver (Go) | Fail-fast circuit breaker for backend-down state | 4.1-4.2 | ApiserverBackendBridge (P0, consumes rejoin/error signals) | State |

### Shared Library

#### ConfigConverter

| Field | Detail |
|-------|--------|
| Intent | Turn a validated `xce_cluster_config_t` into a fully-populated `xce_client_config_t`, clamped to `N_MAX_MASTERS` with a WARN on truncation |
| Requirements | 1.1, 1.2, 1.3, 1.5, 1.6, 1.7, 6.1 |

**Responsibilities & Constraints**
- Pure, socket-free transformation (mirrors `xce_test`'s existing inline logic at `test.c:360-372`, generalized into one reusable function).
- Does not call `xce_cluster_config_validate()` itself — callers validate first (existing contract, unchanged) and pass a pre-validated config.
- Does not own `from`/`key` (identity/auth) — caller sets those on the output struct after conversion, matching how `xce_test`, worker, and scheduler already source them independently (CLI/env-specific).
- **Handles `nmasters=1` as the ordinary case, not a special one** (Req 1 AC5): a one-candidate `cfg` produces a one-master `out` through the identical copy loop as any other count — no branch distinguishes "single controller" from "multi controller" anywhere in this function.
- **The `N_MAX_MASTERS` clamp is now a defensive safety net, not a normally-triggered path** (Req 1 AC7): since `client.h`'s `N_MAX_MASTERS` now derives from `config.h`'s `MAX_HA_CANDIDATES`, and `xce_cluster_config_validate()` already rejects `candidates_count > MAX_HA_CANDIDATES` before a config ever reaches this function, `cfg->candidates_count > N_MAX_MASTERS` cannot occur via the validated-input contract this function documents as its precondition. The clamp+WARN stays in place as a guard against a caller that skips validation or a future re-divergence of the two constants — it is intentionally not removed, just no longer load-bearing in the normal path.

**Dependencies**
- Inbound: WorkerControllerConnector, SchedulerControllerClient, ApiserverBackendBridge (all P0)
- Outbound: none (leaf function)

**Contracts**: Service [x]

##### Service Interface
```c
/* xce/cluster.h */
/**
 * xce_cluster_config_to_client_config() — convert a validated candidate list
 * into an xce_client_config_t's master fields (nmasters/masters/master_ports).
 *
 * Copies up to N_MAX_MASTERS candidates from cfg->candidates[] in order.
 * If cfg->candidates_count > N_MAX_MASTERS, copies the first N_MAX_MASTERS and
 * logs a WARN identifying the truncated count (mirrors xce_test's existing
 * clamp behavior).
 *
 * Does NOT set out->from / out->key — caller sets those separately.
 * Does NOT validate cfg — caller must have already called
 * xce_cluster_config_validate() successfully.
 *
 * Returns 0 on success, -EINVAL if cfg or out is NULL or cfg->candidates_count < 1.
 */
int xce_cluster_config_to_client_config(const xce_cluster_config_t* cfg,
                                        xce_client_config_t* out);
```
- Preconditions: `cfg` already passed `xce_cluster_config_validate()`.
- Postconditions: `out->nmasters` in `[1, N_MAX_MASTERS]`; `out->masters[i]`/`out->master_ports[i]` populated for `i < out->nmasters`, in the same order as `cfg->candidates[]`.
- Invariants: never writes beyond `N_MAX_MASTERS` entries regardless of `cfg->candidates_count`.

### Worker

#### WorkerControllerConnector

| Field | Detail |
|-------|--------|
| Intent | Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect |
| Requirements | 1.1-1.5, 2.1-2.3, 2.5, 3.1, 3.3, 5.1-5.2 |

**Responsibilities & Constraints**
- Replaces `xcecd_join_cluster`'s per-candidate loop (`controller_connector.c:239-312`) with: build ONE `xce_cluster_config_t` — from `cc_ctx->cluster_cfg` when `--peers` was given, or synthesized as a single-candidate `xce_cluster_config_t` (`candidates_count=1`, from the existing hardcoded `"localhost":9990` default, `main.c:333-334`) when it was not — run it through `xce_cluster_config_validate()`, then ConfigConverter, then one `xcecli_ctx_init` + `xcecli_start` + `xcecli_join`. **Deletes `init_client_config()`'s standalone direct-construction of `xce_client_config_t`** (`controller_connector.c:129-160`) and `xcecd_join_cluster()`'s separate "single-host fallback (original path)" branch (`controller_connector.c:297-311`) — both were a wholly parallel mechanism that never touched `xce_cluster_config_t`/`xce_parse_peers`/`xce_cluster_config_validate` at all (confirmed by source audit, 2026-08-10). Per the redefined Req 1 AC5, single-host is not a separately-maintained fallback; it is the `candidates_count=1` input to the exact same synthesize-validate-convert-connect sequence multi-candidate configs use.
- Fixes the init-order bug by moving `cc_ctx->cluster_cfg` population (`main.c`) to before `xcecd_start()` is called, so the very first connect already sees the full candidate list (Req 1.4) — this fix composes naturally with the single-path unification above: since there is no longer a separate "first connect uses the single-host branch" code path to accidentally fall into, the only remaining ordering requirement is that whichever `xce_cluster_config_t` gets built (real `--peers` list or synthesized single-candidate) is fully assembled before the one synthesize-validate-convert-connect sequence runs, on both the first attempt and any later one.
- Registers a rejoin callback (`xcecli_set_reconnect_handler`) that only sets a `needs_rejoin` flag and returns `0` — per "Rejoin callback constraint" above, it must not call `xcecli_join` (or any `xcecli_*` function) directly, since the library invokes it while `ac->failover.lock` is still held.
- `checking_connection()`'s periodic `xcecli_hello` keeps its existing role (liveness probe on the connector thread) but no longer needs to manually call `xcecd_join_cluster` on failure — a transport failure inside `xcecli_hello` already triggers the library's transparent reconnect + the registered flag-setting callback. Immediately after each `xcecli_hello` call returns (lock now free), `checking_connection()` checks `needs_rejoin`; if set, it calls `xcecli_join(ctx, HostRole_worker, &host_attrs)` as a separate call — the same join payload used at startup. **The flag clears only on that join call's success** (Req 2 AC5): if `xcecli_join` itself returns `-E_XCE_NOT_READY` (its own retry budget exhausted), `needs_rejoin` stays set, so the next `xcecli_hello` tick's return retries the join — `checking_connection()`'s existing periodic cadence is what makes this retry naturally, no new timer needed.
- Runs entirely on the existing dedicated connector thread (`start_controller_connector_thread`/`connector_to_controller`), preserving isolation from the main-loop `worker_process_update()` and `resource_sync_thread` liveness paths (Req 3.1) — no new thread introduced.

**Dependencies**
- Inbound: `xcecd_start()` (P0)
- Outbound: ConfigConverter (P0), `xcecli_*` (P0)

**Contracts**: Service [x]

**Implementation Notes**
- Integration: preserve existing WARN-level log statements around standby-redirect and candidate-exhaustion, now emitted from the library's classification path or immediately around the single init/join call, so Req 5 diagnostics are not lost in the simplification.
- Validation: implementation must confirm (code read, not test-only) that the connector thread is isolated from the main-loop `worker_process_update()` and the dedicated `resource_sync_thread` — the two recurring liveness paths — flagged as a Validation Hook, not re-verified in this design pass. (The three `*_update` calls previously cited — `hardware_monitor_update`/`system_monitor_update`/`process_manager_update` — are one-shot init calls at `xcecd.c:476-480` on the main thread, not recurring liveness duties; corrected per 2026-08-10 gap re-validation.)
- Risks: deleting the hand-rolled loop must not silently drop the `MAX_RETRY_COUNT`/`RETRY_INTERVAL` bounded-retry-then-`handle_disconnect()` behavior in `checking_connection()` — that outer bound (Req 2.3's "keep retrying" vs. any existing bounded-give-up behavior) must be reconciled explicitly during implementation, not left ambiguous.

### Scheduler

#### SchedulerControllerClient

| Field | Detail |
|-------|--------|
| Intent | Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect |
| Requirements | 1.1-1.3, 1.5-1.7, 2.1-2.3, 2.5, 3.2-3.3, 5.1-5.2, 7.1-7.6 |

**Responsibilities & Constraints**
- Replaces `sched_init`'s inline per-candidate loop (`scheduler.c:564-624`) with ConfigConverter + one `xcecli_ctx_init`/`xcecli_start`/`xcecli_join` call. **Deletes `sched_init`'s "single-host path (original)" `else` branch** (`scheduler.c:613`, confirmed by source audit 2026-08-10 to be a wholly separate direct-construction mechanism when `--peers` is not given) — when no `--peers` is given, `sched_init` now synthesizes a single-candidate `xce_cluster_config_t` (`candidates_count=1`, from `init_config()`'s existing `"localhost":9990` default, `main.c:143-152`) and runs it through the identical `xce_cluster_config_validate()` → ConfigConverter → connect sequence as any real `--peers` list, per the redefined Req 1 AC5: single controller is `candidates_count=1` of the one mechanism, not a parallel fallback.
- `--sim` mode early-returns before `sched_init()` (`main.c:296`), so the peers-parsing block only reaches `sched_init()` for normal-mode operation — the design's assumption that candidate-list wiring always runs before `sched_init()` in the normal-mode path is confirmed correct; only the line numbers shifted.
- Registers a rejoin callback that only sets a `needs_rejoin` flag and returns `0` (same constraint as worker — the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join` itself). **Corrected call site** (supersedes an earlier, incorrect claim that this runs on `sched_allocator`): `xcecli_jobdispatch` is called from `sched_publish_dispatch` (`scheduler.c:1257`), which is reached only via the decision-drain loop in `xce_sched_driver_run_normal_mode_round` (`driver.c:765-786`) — a loop that runs on the single main-loop thread (`main.c`'s `while(1)`, `main.c:348-349`), not on `sched_allocator`. `sched_allocator` only pushes decisions onto the `decisions` cbuf (`scheduler.c:336,387`); it never calls `xcecli_jobdispatch` directly. The check-`needs_rejoin`-then-`xcecli_join` call must therefore live in `sched_publish_dispatch`, immediately after its `xcecli_jobdispatch` call returns (lock released) — same thread, same call frame — **not** in `sched_allocator`. **The flag clears only once that `xcecli_join` call succeeds** (Req 2 AC5): on its own `-E_XCE_NOT_READY`, `needs_rejoin` stays set, so the next decision's publish-return retries the join — since Req 7's retry-until-success wrapper means `sched_publish_dispatch` returns (successfully) far more reliably now than before, this composes cleanly rather than needing its own separate retry loop.
- **Corrected thread/backpressure model** (Critical Issue from 2026-08-10 design validation, resolved by narrowing below — supersedes the prior claim that `sched_job_picker` runs "unaffected"): the scheduler's real pipeline is three stages joined by two `cbuf`s: `sched_job_picker` → `jbuf` (capacity 1, `core.c:161` — the scheduler-core determinism invariant, never resizable without re-deriving `scheduler-core`'s Req 5 AC7) → `sched_allocator` → `decisions` (capacity `ndecisions`=5, `core.c:84`) → the main-loop thread's decision-drain loop → `xcecli_jobdispatch`. When that call blocks inside `XCECLI_FAILOVER_CALL`'s retry/backoff (lock held), the main thread stops draining `decisions`. `sched_allocator` can still produce more decisions before its own `cbuf_put(&core.decisions, ...)` blocks, and `sched_job_picker` can still produce jobs into `jbuf` for as long as allocator keeps draining it. So picker and allocator are **buffered, not isolated**: they continue briefly past the point the outage begins, then both stall until connectivity (and dispatch) resumes. **The exact size of that grace window is occupancy-dependent, not a fixed constant**: if `decisions` had Q items queued (1≤Q≤5) at the moment the main thread's pop grabs the decision that turns out to be un-publishable, allocator can push (6-Q) more decisions before it stalls, and picker's total new picks during the window come out to (7-Q) — ranging from 2 (Q=5, `decisions` was already near-full, common under load) up to 6 (Q=1, `decisions` was nearly empty). Earlier text in this section claimed "at least 6" as a guaranteed floor — that was backwards: 6 is the best-case ceiling (`ndecisions`+1), the guaranteed floor is as low as 1-2 depending on `jbuf`'s own occupancy at the same instant. This is a materially different, weaker guarantee than "unaffected," and a materially different, more accurate one than the original "at least 6" claim. This "stall until resumes" description is only actually true once the Req 7 fix below is in place — see that bullet for why, unfixed, the main thread can instead silently unstick itself by dropping a decision rather than genuinely staying stalled.
- **Decision-publish retry-until-success-or-shutdown wrapper (Req 7, gap found in 2026-08-10 design validation)**: `xcecli_jobdispatch`/`xcecli_set_pend_reason` are `XCECLI_FAILOVER_CALL`-wrapped, which retries a *single call* up to a bounded budget (`max_retries`=20, backoff capped at `max_retry_ms`=5000ms — roughly 1-2 minutes total, `client_failover.c:281-282,312-317`) before returning `-E_XCE_NOT_READY`. Today's driver decision-drain loop (`driver.c:783-789`) does not inspect this return code — it logs and advances to the next queued decision regardless of outcome. Since `sched_commit_decision` already mutates scheduler-local job state (`job->allocs`, accounts) *before* the publish attempt (`scheduler.c` `sched_allocator`), a budget-exhausted publish today silently and permanently diverges scheduler-local state from what the controller was told — not a delay, a loss. Fix: `sched_publish_dispatch`/`sched_publish_pend_reason` (`scheduler.c:1257`/`1251`) wrap their `xcecli_*` call in `while (!sched->core.stop) { rc = XCECLI_FAILOVER_CALL(...); if (rc != -E_XCE_NOT_READY) break; WARN(extended-retry, ...); }` — checking `core.stop` **before** each attempt, not only after (a plain `do`/`while` checking the stop condition only at the bottom would let every already-queued decision behind the one currently stuck each burn one full ~100s budget cycle discovering shutdown was requested, compounding to minutes-per-queued-decision instead of a single bounded delay — caught in the 2026-08-10 design validation's third pass). Re-invoking the whole macro on each loop iteration resets its own internal retry state and starts a fresh budget, rather than adding a second retry mechanism. This is a straightforward extension of Req 2 AC3's existing "keep retrying, never permanently give up" language, applied per-decision instead of only at the connection level. **Shutdown interaction**: `stop_scheduler()`'s existing `cbuf_tryput(&core.decisions, NULL)` wake-up (`main.c:163`) only unblocks a thread parked in `cbuf_get(&core.decisions)` — it does nothing for a thread already inside this new retry loop, so the loop's own `core.stop` check is the only shutdown path. With the check moved to the top, only the ONE decision actively in-flight when shutdown is requested can cost up to one full budget cycle (~1-2 minutes); every decision still queued behind it in `decisions` sees `core.stop` already true and returns immediately without attempting a call, so total shutdown latency stays bounded to a single cycle regardless of queue depth — an accepted bounded tradeoff (same posture as scheduler-core's accepted 5s `OpCode_schedAdmin` blocking-wait case) since `libxce`'s retry macro itself is out of this spec's boundary to modify for a faster mid-attempt shutdown check. **Ordering**: decisions remain published strictly in `decisions` cbuf FIFO order (Req 7 AC3) because the drain loop still only advances to `cbuf_get()` the next decision after the current one's publish call returns — a stuck decision blocks the whole loop, it does not get skipped and revisited later. **Rejoin-check interaction**: the existing check-`needs_rejoin` step (Responsibilities bullet above) sits after this whole `while` loop, not after each inner attempt — it observes the loop's final, successful return, exactly the same as it would for a single non-retried call.

**Dependencies**
- Inbound: `sched_init()` (P0)
- Outbound: ConfigConverter (P0), `xcecli_*` (P0)

**Contracts**: Service [x]

**Implementation Notes**
- Integration: preserve the existing `WARN`/`INFO` log lines around candidate parsing (`main.c`) and add per-attempt/per-outcome logging around the single init/join call (today's loop logged each attempt; the library's own classification path should be checked for equivalent visibility, else add it locally).
- **Req 3 AC2 scope note (Critical Issue from 2026-08-10 design validation, resolved 2026-08-10)**: the original merged AC1 wording ("shall continue performing ... allocation/decision processing ... without being blocked") was not met unconditionally by the scheduler — only for a bounded, occupancy-dependent grace window (floor as low as 1-2 jobs, ceiling `ndecisions`+1=6, see the corrected backpressure-model bullet above), after which `sched_allocator` and then `sched_job_picker` do block. `jbuf`'s capacity is a fixed invariant owned by the `scheduler-core` spec (out of this spec's boundary to change) and `decisions`' capacity is an existing config constant; redesigning either is out of scope here. Resolution: Requirement 3's single merged AC1 was split into a worker-only AC1 (unconditional, unchanged guarantee) and a scheduler-only AC2 (bounded — "continues briefly ... then blocks until connectivity is restored, at which point processing resumes automatically ... no job picked or decided during that bounded window is ever lost"), with the former heartbeat AC renumbered to AC3. `requirements.md` deliberately does not hardcode a specific job count — see the corrected backpressure-model bullet above for why that count is occupancy-dependent (floor 1-2, ceiling `ndecisions`+1=6), not a fixed constant an initial draft of this note incorrectly stated as a guaranteed "at least 6." `requirements.md` reflects this split and the corrected framing; the design and its bounded-buffering behavior now match the requirement it's held to, with Requirement 7 supplying the actual load-bearing guarantee (no loss, regardless of how many jobs the buffer absorbs).
- Risks: today's total absence of reconnect logic means this is additive behavior change (scheduler previously required a manual restart after primary loss) — call out explicitly in the task/rollout notes so operators aren't surprised by scheduler now retrying indefinitely instead of exiting.

### apiserver

#### ApiserverBackendBridge

| Field | Detail |
|-------|--------|
| Intent | Single connect/join at startup using the full candidate list via the C library (through CGO), replacing the Go-level per-peer loop; config-syntax parity with scheduler/worker |
| Requirements | 1.1-1.7, 2.4-2.5, 5.1-5.2, 6.1-6.2 |

**Responsibilities & Constraints**
- `RegisterToBackend()` (`clib.go`) is rewritten to build exactly ONE C `xce_cluster_config_t` regardless of source, then run it through the single synthesize-validate-convert-connect sequence worker/scheduler use: if `config.Peers` is non-empty, format it into the same `[name=]host:port[,...]` syntax and parse via `xce_parse_peers` (CGO, not the existing hand-written Go `parsePeer`); if empty, synthesize a one-candidate `xce_cluster_config_t` (`candidates_count=1`) directly from `config.Address`/`config.Port` — **either way** the result is validated with `xce_cluster_config_validate`, converted with ConfigConverter, then `xce_backend_init` is called **once** with the resulting config (`nmasters` in `[1, N_MAX_MASTERS]`, not necessarily `>1`). **Deletes `tryConnect()`'s direct `C.xce_client_config_t` construction** (`clib.go:79-101`) and the `for _, peer := range config.Peers { tryConnect(...) }` Go-level loop (`clib.go:116-135`) entirely, along with `parsePeer()` (`backend/peers.go:11`, retired per Req 6.1) — both `tryConnect()`'s Peers-loop call site and its no-Peers `Address`/`Port` call site (`clib.go:138`) were separate mechanisms that never touched `xce_cluster_config_t` at all (confirmed by source audit, 2026-08-10); per the redefined Req 1 AC5, a bare `Address`/`Port` config is `candidates_count=1` of the same mechanism, not a separate fallback.
- `reconnect.go` holds two plain Go functions: (1) a `//export`-annotated flag-setter — registered as the library's rejoin callback, does no I/O, only sets a `needs_rejoin` atomic flag and returns `0` (per "Rejoin callback constraint" above: the library invokes it while `ac->failover.lock` is held, so it must not call `xcecli_join`/any CGO backend function itself); (2) a plain Go function that performs the actual rejoin — calls `xcecli_join(gCtx.ctx, HostRole_apiserver, NULL)` and clears `ApiserverBackendHealth`'s degraded state on success.
- The flag-setter is **registered with the C library from `clib.go`'s own cgo preamble**, via a small static C shim (`xce_backend_rejoin_handler`, mirroring the existing `xce_backend_handler`/`go_push_handler` pattern already used for push dispatch) passed to `xcecli_set_reconnect_handler` — cgo symbol resolution is per-file (see `wire_dispatch_support.go`'s documented rationale for the identical `go_push_handler` wrapper), so the shim and the registration call must live in the same file as the `extern` declaration, not standalone in `reconnect.go`.
- Rather than duplicating the same pre-call/post-call logic across all four wrapper functions, the pre-call `IsDegraded()` fail-fast check (ApiserverBackendHealth, below) and the post-call `needs_rejoin` check are both implemented once, in a single shared internal helper each of the four `clib.go` wrappers calls around its `C.xcecli_*` invocation — not copy-pasted per wrapper. **The flag clears only once the follow-up `xcecli_join` call succeeds** (Req 2 AC5): if that call itself returns `-E_XCE_NOT_READY`, `needs_rejoin` stays set so the next backend-call wrapper's return retries the join, and `MarkHealthy()` is correspondingly only called on that eventual success — not on the mere attempt.
- **Idle-period reconnect gap (Req 2.4) fix**: `XCECLI_FF_CALL` — the macro backing `xcecli_heartbeat`, `heartbeat.go`'s existing periodic prober (`cmd/main.go:76`) — performs a best-effort transport reconnect on a known-dead connection but never invokes the library's rejoin hook (verified against `client_internal.h`/`client_failover.c`: `xcecli_invoke_rejoin` is called only from `xcecli_retry_next`, the full-retry path used by write calls, never from the fire-and-forget path). Left as-is, a controller failover occurring while apiserver has no in-flight write request would silently repair the TCP connection without ever re-joining, permanently starving push notifications until an unrelated write request happens to arrive. Fix: on a failure→success transport transition, `heartbeat.go`'s tick callback sets the **same `needs_rejoin` atomic flag the write-path's C-side callback sets** — since `XCECLI_FF_CALL` never invokes that callback for this path, `heartbeat.go` sets the Go-side flag itself. On every subsequent tick where `xcecli_heartbeat` succeeds **and `needs_rejoin` is still true** (checked every tick, not only on the transition edge — Req 2 AC5 gap found in the 2026-08-10 design validation's second pass), it calls `reconnect.go`'s rejoin function; the flag (and `ApiserverBackendHealth`'s degraded state) clears only when that rejoin call itself succeeds. **Deliberately not gated on `ApiserverBackendHealth.IsDegraded()`** (a design-validation third-pass correction over an earlier draft of this fix): that state has its own independent TTL-based auto-clear (see ApiserverBackendHealth below) for request-path circuit-breaker purposes — if the TTL expired before the rejoin actually succeeded, gating the retry on "still degraded" would stop retrying the join even though it never succeeded, silently reintroducing the same starvation this fix exists to close. `needs_rejoin` and `ApiserverBackendHealth`'s degraded-until timestamp are two independent pieces of state serving two different purposes (one tracks "have I re-joined yet," the other "should request handlers fail fast right now") and must not be conflated. A transition-edge-only trigger would fire the rejoin attempt exactly once per outage and, if that one attempt's own `xcecli_join` hit its retry budget and failed, never retry — every subsequent tick would see success→success (transport already up) and never re-fire; checking the persistent flag instead of the transient edge makes each tick a natural retry opportunity for as long as apiserver remains un-joined. This reuses the same rejoin logic Task 4.3 already builds — no duplicate join-payload code.

**Dependencies**
- Inbound: `cmd/main.go` (P0)
- Outbound: ConfigConverter via CGO (P0), `xcecli_*` via CGO (P0), ApiserverBackendHealth (P1, signals it on outcome)

**Contracts**: Service [x]

##### Service Interface
```go
// backend/clib.go
// RegisterToBackend initialises the client using the full configured
// candidate list — config.Peers if set, otherwise a single-candidate list
// synthesized from config.Address/config.Port (not a separate code path,
// see Req 1 AC5) — and registers the rejoin callback. Single call — no
// per-peer Go loop.
func RegisterToBackend(config *BackendConfig) error
```
- Preconditions: `StartNotificationListener` already called (unchanged from today).
- Postconditions: on success, `gCtx.ctx` is joined with `nmasters` reflecting every configured candidate (clamped per ConfigConverter); the rejoin callback is registered.
- Invariants: candidate-list parsing/validation errors surface the same way `xce_parse_peers`/`xce_cluster_config_validate` report them in C (Req 6.2) — not a separately-worded Go error.

**Implementation Notes**
- Integration: `parsePeer` (Go) and its tests are removed/retired in favor of the CGO-routed parser (Req 6.1).
- Integration: the rejoin C shim must be defined in `clib.go`'s cgo preamble (not `reconnect.go`) — see Responsibilities above; this was the design's one build-risk item, now resolved by construction rather than left as an implementation-time surprise.
- Risks: CGO error surfaces (C `-EINVAL` etc.) need a clear Go-side error message mapping so operators get an actionable message (Req 1.6, 6.2), not a bare error code.

#### ApiserverBackendHealth

| Field | Detail |
|-------|--------|
| Intent | Let request handlers fail fast (503) instead of blocking on the library's full retry budget when the backend is known-degraded |
| Requirements | 4.1, 4.2 |

**Responsibilities & Constraints**
- New `health.go`: an atomic "degraded-until" timestamp. `IsDegraded() bool` — true if `now < degraded-until`. `MarkDegraded(ttl time.Duration)` — called by backend-call wrapper functions in `clib.go` when a call's returned error indicates retry-budget exhaustion (e.g. `E_XCE_NOT_READY`). `MarkHealthy()` — called by the rejoin callback (`reconnect.go`) on a successful rejoin.
- Backend-call wrapper functions used by request-handling code paths (job submit, job control, queue control, host control, etc. — the existing `gCtx.ctx`-based calls in `clib.go`) check `IsDegraded()` first, via the shared internal helper described under ApiserverBackendBridge above (not duplicated per wrapper); if true, return a clear "backend unavailable" error immediately instead of invoking the blocking C call (Req 4.1). Calls that don't require the backend are unaffected (Req 4.2 — nothing to change there, they already don't touch `gCtx`).

**Dependencies**
- Inbound: ApiserverBackendBridge (sets state on outcomes)
- Outbound: none

**Contracts**: State [x]

##### State Management
- State model: single atomic degraded-until timestamp (int64 unix-nano via `sync/atomic`, same pattern as existing `heartbeatIntervalSecs` in `heartbeat.go`).
- Persistence & consistency: process-local, in-memory only; reset on apiserver restart (matches the rest of apiserver's in-memory state model per `tech.md`).
- Concurrency strategy: lock-free atomic read/write; safe for concurrent goroutines per request.

**Implementation Notes**
- Integration: TTL-based clearing (Risks & Mitigations in research.md) in addition to the rejoin-callback clear, so a missed/edge-case rejoin doesn't permanently wedge the flag.
- Validation: unit tests cover degraded→healthy transitions via both the TTL expiry and explicit `MarkHealthy()`.
- **Deliberately independent of `needs_rejoin`** (Req 2 AC5, correction from the 2026-08-10 design validation's third pass): `IsDegraded()`'s TTL-based auto-clear exists so request handlers eventually probe again even without an explicit `MarkHealthy()` — a legitimate circuit-breaker half-open pattern. It must **not** be used as the retry-trigger condition for the heartbeat-idle-path's rejoin attempts (ApiserverBackendBridge above) — the TTL could expire before an actual rejoin succeeds, and gating the rejoin retry on "still degraded" would then silently stop retrying the join while apiserver is still not actually joined. The rejoin-tracking `needs_rejoin` flag has no TTL and clears only on an actual `xcecli_join` success, independent of whatever `ApiserverBackendHealth`'s degraded-until timestamp is doing.

## Error Handling

### Error Strategy
No new error taxonomy — reuse the existing controller status codes (`E_XCE_REDIRECT`/`E_XCE_STANDBY`/`E_XCE_NOT_READY`, `errnox.h`) that `client_failover.c` already classifies, and the existing `xce_cluster_config_validate()` `-EINVAL` cases for malformed candidate-list config.

### Error Categories and Responses
- **Malformed candidate-list config** (Req 1.6, 6.2): `xce_parse_peers`/`xce_cluster_config_validate` failure at startup → each component reports a clear message naming the offending entry and refuses to start (unchanged behavior for worker/scheduler; apiserver gains the same via the CGO route).
- **All candidates unreachable at startup** (Req 1.6): `xcecli_start`/`xcecli_join` returns the last transport error after exhausting all candidates → component reports which candidates were attempted (Req 5.2) and fails startup (worker/scheduler: process exits as today; apiserver: `RegisterToBackend` returns an error, `cmd/main.go`'s existing handling applies unchanged).
- **Runtime disconnect** (Req 2): transparently retried/rotated by the library; components only observe it through their own periodic calls' return codes and the rejoin callback — no new error category, just new logging around the existing outcomes.
- **Scheduler decision-publish retry-budget exhaustion** (Req 7): `-E_XCE_NOT_READY` from `xcecli_jobdispatch`/`xcecli_set_pend_reason` is not treated as terminal — the caller re-invokes the same publish call rather than surfacing an error or dropping the decision (see SchedulerControllerClient's retry-until-success-or-shutdown wrapper). This is scheduler-specific: worker's periodic `xcecli_hello` carries no payload to lose on the same error, and apiserver's request/response calls correctly surface `-E_XCE_NOT_READY` to `ApiserverBackendHealth` since there the caller (the original HTTP client) is expected to retry, unlike scheduler's decisions which have no external retry source.
- **Candidate count exceeds `N_MAX_MASTERS`**: ConfigConverter clamps and WARNs (not an error) — matches `xce_test`'s existing behavior exactly.
- **apiserver backend degraded** (Req 4.1): mapped to a clear 503-style API error, distinct from other 5xx causes, so operators/clients can distinguish "backend down, retrying" from an unrelated internal error.

### Monitoring
Per-candidate connect/reconnect attempts and outcomes are logged (Req 5) using each component's existing logger (`XCECD_LOG` for worker, the scheduler's existing `INFO`/`WARN` macros, `zap` for apiserver) — no new logging infrastructure, just new call sites around the simplified connect/reconnect path.

## Testing Strategy

- **Unit Tests**:
  - `ConfigConverter`: clamps at `N_MAX_MASTERS` with WARN when given more candidates; preserves candidate order; rejects `NULL`/empty input (`test_cluster.c`); a `candidates_count=1` input produces `nmasters=1` through the identical code path as any other count (Req 1.5) — no branch, no special case.
  - `N_MAX_MASTERS == MAX_HA_CANDIDATES` (Req 1.7): a `_Static_assert` (or equivalent compile-time check) in `client.h` or a `test_cluster.c` case asserting the two constants stay equal — a regression guard against reintroducing the divergence this design pass reconciled. Also re-run `test_libxce_retry.c`'s `fo_make_client_masters()` fixture with its corrected slot count and confirm it doesn't silently exercise an uninitialized `masters[3]`.
  - apiserver `health.go`: degraded→healthy via `MarkHealthy()`, via TTL expiry; concurrent goroutine safety (`health_test.go`).
  - apiserver CGO-routed parser: same malformed-input cases scheduler/worker already reject, now asserted through the Go call site (`clib_test.go`).
  - Rejoin flag-setter callbacks (worker, scheduler, apiserver): with a real `ac->failover.lock` held (simulating the library's call context per "Rejoin callback constraint"), invoking the registered callback returns promptly without blocking and performs no I/O — a regression guard against reintroducing the self-deadlock this design pass found.
  - Rejoin retry-on-failure (Req 2 AC5, worker/scheduler/apiserver): using a test double for `xcecli_join` that fails N times before succeeding, assert the pending-rejoin indicator (flag or degraded-state) stays set/degraded after each failed attempt and is cleared only on the eventual success; assert the next naturally-occurring opportunity (next `xcecli_hello`/decision-publish/backend-call return, or next heartbeat tick for apiserver's idle path) is what re-attempts it, not a dedicated new retry loop — a regression guard against the transition-edge/clear-before-success versions of this bug found in the 2026-08-10 design validation's second pass.
- **Integration Tests**:
  - Scheduler decision-publish redelivery (Req 7): using a test double for `xcecli_jobdispatch`/`xcecli_set_pend_reason` that returns `-E_XCE_NOT_READY` for N calls before succeeding, assert the same decision is retried (not dropped, not skipped) and eventually published; assert decisions queued behind it publish afterward in original order (Req 7 AC1-3); assert setting `sched->core.stop` mid-retry breaks the loop within one budget cycle (Req 7 AC2); assert a WARN-level log line distinguishes extended-retry success from first-attempt success (Req 7 AC4); with multiple decisions queued (`decisions` at or near its 5-item capacity) when `core.stop` is set mid-retry on the first, assert total shutdown latency stays bounded to roughly one budget cycle, not one per queued decision — a regression guard against the compounding-shutdown-latency bug found in the 2026-08-10 design validation's third pass.
  - Unified single-host path (Req 1.5): starting each of worker, scheduler, and apiserver with no `--peers`/empty `backend.peers` reaches the identical `xce_cluster_config_validate` → ConfigConverter → connect call sequence as a single-entry `--peers` config — assert via the same log-line signature both paths now share (there is no longer a distinct "single-host mode" log message), not just "it connects."
  - Worker/scheduler startup against a 3-candidate config where candidate[0] is a live standby — assert the component joins as expected without operator intervention (Req 1.1-1.3), extending each component's `testsuite/`.
  - Worker/scheduler reconnect: kill the currently-joined primary mid-session, assert the component reconnects and re-joins the new primary without restart (Req 2.1-2.3), reusing the project's existing `/xce-deploy-ha` + `/xce-test-ha` pattern already used for controller HA soak testing — this is also the test that would hang (not fail cleanly) if the self-deadlock were reintroduced, so it must run with a wall-clock timeout that fails loud rather than hanging CI.
- **E2E Tests**:
  - Full 3-node HA cluster with worker + scheduler + apiserver all configured with the full candidate list; kill the primary; assert all three reconnect, worker resumes job execution reporting, scheduler resumes dispatch, apiserver resumes serving requests and receiving push notifications (Req 2.4) — new `test_e2e_multimaster_join.c` per C component plus an apiserver-side check in the existing HA soak harness.
  - apiserver idle-period reconnect: kill the primary while apiserver has no in-flight write request (only the heartbeat prober running); assert apiserver still rejoins and resumes push notifications, proving the heartbeat-triggered rejoin path (not just the write-call-triggered one) works (Req 2.4).
  - apiserver fail-fast: with the backend killed and not yet reconnected, assert an API request returns quickly (bounded, sub-second) with a clear 503 rather than hanging for the full retry budget (Req 4.1).

## Security Considerations
No new trust boundary: candidate hosts are the same set of controllers already trusted today; TLS/identity handling is unchanged (see Non-Goals — CA-cert wiring is out of scope and pre-existing). No new authentication surface is introduced.