myxce/.kiro/specs/component-failover-adoption/design.md

Design Document: component-failover-adoption

Overview

Scheduler, worker, and apiserver each connect to the controller cluster today using only a single configured endpoint at a time, with divergent, partly-broken hand-rolled logic for trying alternates. Meanwhile libxce already ships a complete, tested, transparent multi-master mechanism (XCECLI_FAILOVER_CALL and friends in src/lib/client.c) — the same one xce_test's masters setting already uses correctly. This design wires all three components onto that existing mechanism: populate the full candidate list once, let the library rotate/retry/reconnect on every call, and register a rejoin callback so a component re-establishes cluster membership after a transparent reconnect.

Users: HA cluster operators who configure multiple controller candidates (--peers for scheduler/worker, backend.peers YAML for apiserver) and expect scheduler, worker, and apiserver to survive a controller failover without manual restarts — the same expectation xce_test already meets.

Impact: Removes worker's and scheduler's hand-rolled per-candidate join loops (deleting, not adding, most of the connection-retry code in controller_connector.c and scheduler.c); replaces apiserver's Go-level per-peer loop with a single library-driven connect; adds one new small shared C helper and a small Go-side health guard. No changes to the controller or to libxce's failover mechanism itself.

Goals

• Scheduler, worker, and apiserver each hand the full configured candidate list to xce_client_config_t on the very first connection attempt (Req 1).
• All three automatically reconnect and resume cluster membership after their controller connection is lost, without a process restart (Req 2).
• Worker's and scheduler's retry/backoff never blocks duties independent of controller connectivity (Req 3).
• apiserver returns a clear, fast error to its own API clients while its backend connection is down, instead of hanging (Req 4).
• Every connect/reconnect attempt and outcome is logged per-candidate (Req 5).
• apiserver accepts the identical candidate-list syntax scheduler/worker already accept (Req 6).

Non-Goals

• Changing xce_test (it is already the reference-correct client).
• Completing or adopting the separate, unused xce_connect_cluster()/xce_reconnect_* primitive in cluster.c — superseded by the mechanism this design adopts; left untouched.
• Raising the N_MAX_MASTERS (3) cap or reconciling it with MAX_HA_CANDIDATES (4) — pre-existing constraint, replicated (clamp + WARN) exactly as xce_test already does.
• Wiring CA-cert/TLS trust into the real connect path — xce_client_config_t carries no such field today and this feature does not add one; TLS is established the same way it already is.
• Controller-side changes (redirect-reply population, peer authentication) — already shipped by libxce-failover.
• Recovering worker job-state lost during a disconnect window — owned by worker-rejoin-resync.

Boundary Commitments

This Spec Owns

• A new shared libxce helper that converts a validated xce_cluster_config_t (parsed candidate list) into a fully-populated xce_client_config_t.
• Worker's, scheduler's, and apiserver's integration code that calls xcecli_ctx_init/xcecli_start/xcecli_join with the full candidate list and registers a rejoin callback.
• apiserver's candidate-list config parsing (routed through the existing C parser via CGO instead of a second Go implementation).
• apiserver's client-facing fail-fast behavior while its backend connection is degraded.
• Connect/reconnect diagnostic logging in all three components.

Out of Boundary

• xce_test and its masters setting (unchanged, reference implementation).
• libxce's failover mechanism itself (client.c, client_internal.h, client_failover.c) — consumed as-is, not modified.
• xce_connect_cluster()/xce_reconnect_* (unused, untouched).
• Controller-side membership/redirect logic.
• Worker job-state resync content on rejoin (worker-rejoin-resync).
• N_MAX_MASTERS/MAX_HA_CANDIDATES value changes.
• TLS/CA-cert plumbing into the real connect path.

Allowed Dependencies

• src/lib/client.c / client_internal.h / client_failover.h (shipped libxce-failover mechanism) — read-only dependency, no modification.
• src/lib/cluster.c / xce/cluster.h (xce_parse_peers, xce_cluster_config_validate) — existing, unmodified; this spec adds one new function alongside them.
• Existing per-component candidate-list CLI/YAML surfaces (--peers, --ca-cert, backend.peers) — unmodified externally; only how they're consumed changes.

Revalidation Triggers

• Any change to xce_client_config_t's layout or N_MAX_MASTERS.
• Any change to XCECLI_FAILOVER_CALL/XCECLI_FF_CALL retry/classification semantics (libxce-failover).
• Any change to xce_parse_peers/xce_cluster_config_validate syntax or validation rules (apiserver's CGO-based parser would need re-verification).
• worker-rejoin-resync landing — worker's rejoin callback becomes the natural place to also trigger job-state resync; this spec's callback is a no-op-beyond-rejoin placeholder for that.

Architecture

Existing Architecture Analysis

Today each of the three components independently drives its own xcecli_ctx_t lifecycle with nmasters hardcoded to 1 per connection attempt, and layers its own outer candidate loop on top:

• worker: controller_connector.c's xcecd_join_cluster loops candidates itself, but the very first connection (inside xcecd_start()) happens before cc_ctx->cluster_cfg is even populated (main.c assigns it after xcecd_start() returns) — the candidate list is silently ignored on startup.
• scheduler: scheduler.c's sched_init loops candidates correctly at startup, but there is no reconnect logic anywhere afterward.
• apiserver: clib.go's RegisterToBackend loops candidates once via a hand-written Go parser, only at process startup.

None of the three ever set nmasters > 1 on a single xce_client_config_t, so the library's own rotation/retry (client.c, wrapped by XCECLI_FAILOVER_CALL) never actually engages for them.

Architecture Pattern & Boundary Map

graph TB
    subgraph Daemons
        Worker[worker xcecd]
        Scheduler[xce_scheduler]
        Apiserver[apiserver backend package]
    end

    subgraph NewSharedCode
        Convert[xce_cluster_config_to_client_config]
    end

    subgraph ExistingLibxce
        Parse[xce_parse_peers]
        Validate[xce_cluster_config_validate]
        CtxInit[xcecli_ctx_init xcecli_start xcecli_join]
        Failover[XCECLI_FAILOVER_CALL rotate retry backoff]
        Rejoin[rejoin callback xcecli_set_reconnect_handler]
    end

    subgraph Cluster
        Primary[Controller Primary]
        Standby1[Controller Standby]
    end

    Worker --> Parse
    Scheduler --> Parse
    Apiserver --> Parse
    Parse --> Validate
    Validate --> Convert
    Convert --> CtxInit
    CtxInit --> Failover
    Failover --> Rejoin
    Failover --> Primary
    Failover --> Standby1

Architecture Integration:

• Selected pattern: thin per-component integration layer over an unchanged, already-shipped library mechanism (adapter, not new subsystem).
• Domain boundaries: libxce owns candidate parsing, validation, connect/rotate/retry/backoff, and rejoin-hook invocation (unchanged). Each daemon owns only: building the candidate list from its own config surface, calling the library once, and supplying its role-specific rejoin behavior (what to send when re-joining).
• Existing patterns preserved: the xce_parse_peers/xce_cluster_config_validate/xce_client_config_t split; the CGO bridge pattern already used throughout backend/; worker's dedicated connector thread.
• New components: one C helper (candidate-list → client-config conversion, generalized out of xce_test's inline logic) and one small Go health-state module (apiserver-only, has no C-side equivalent need).
• Steering compliance: no new blocking work introduced on iomgr/delaymgr-equivalent threads; worker's existing thread isolation is preserved, not altered.

Rejoin callback constraint: avoiding self-deadlock on ac->failover.lock

Verified directly against client_failover.c:341-411: XCECLI_FAILOVER_CALL holds ac->failover.lock (a plain, non-recursive pthread_mutex_t — mylock_init calls pthread_mutex_init(l, NULL), no PTHREAD_MUTEX_RECURSIVE attribute) for the entire retry loop, and xcecli_invoke_rejoin() — which calls the registered rejoin callback — is invoked synchronously from inside that loop, while the lock is held (client_failover.c:397). Any rejoin callback that itself calls a public xcecli_* wrapper on the same xcecli_ctx_t (e.g. xcecli_join, which is itself XCECLI_FAILOVER_CALL-wrapped and tries to re-lock the same mutex) self-deadlocks — the calling thread blocks forever on its own already-held lock. This was never caught because xcecli_set_reconnect_handler has zero production callers today and the existing unit test's callback is a no-op probe that never calls back into the client.

This directly constrains every component's rejoin implementation in this spec:

• The callback registered with the library (worker's checking_connection→xcecli_hello path, scheduler's xcecli_jobdispatch path, apiserver's write-call path) must do no I/O and call no xcecli_* function — it may only set a lightweight "needs rejoin" flag (a plain bool/atomic on the consumer side) and return 0 so the interrupted operation resumes normally.
• Each component's own calling code, after the triggering xcecli_* call returns (at which point the library has released the lock), checks and clears that flag and issues xcecli_join(...) as a separate, subsequent call — safe, since the lock is free by then.
• apiserver's heartbeat-triggered path (see below) is unaffected by this constraint — it already detects recovery and calls the rejoin logic after xcecli_heartbeat has returned (XCECLI_FF_CALL also releases its lock before returning), so it was never at risk.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Shared library	C (gnu17), src/lib/cluster.c	New candidate→client-config conversion helper	Extends existing file alongside xce_parse_peers
Worker / Scheduler	C (gnu17)	Consume the existing xcecli_ctx_t mechanism directly	No new dependency
apiserver	Go 1.23.4 + CGO	Consume the same mechanism via the existing CGO bridge; new Go-side health-state guard	sync/atomic, consistent with existing heartbeat.go pattern

File Structure Plan

Modified / New Files

src/include/xce/cluster.h                          # + declare xce_cluster_config_to_client_config()
src/lib/cluster.c                                    # + implement xce_cluster_config_to_client_config()
src/lib/testsuite/test_cluster.c                      # + unit tests for the new helper

src/worker/src/controller_connector.c                # rewrite: delete per-candidate loop in
                                                       #   xcecd_join_cluster/init_client_config;
                                                       #   single init+start+join using full
                                                       #   candidate list; register rejoin callback
src/worker/src/main.c                                 # fix: populate cc_ctx->cluster_cfg BEFORE
                                                       #   xcecd_start() (removes init-order bug)
src/worker/include/controller_connector.h             # + rejoin callback function declaration

src/scheduler/src/scheduler.c                         # rewrite: sched_init uses the new helper +
                                                       #   single init+start+join; delete inline
                                                       #   per-candidate loop; register rejoin callback
src/scheduler/src/main.c                              # no ordering change needed (already correct);
                                                       #   simplified call site only

src/frontend/apiserver/backend/clib.go                # rewrite RegisterToBackend: single connect
                                                       #   using the full candidate list via CGO calls
                                                       #   into xce_parse_peers/xce_cluster_config_validate/
                                                       #   xce_cluster_config_to_client_config; cgo
                                                       #   preamble gains a static C shim (mirrors the
                                                       #   existing xce_backend_handler/go_push_handler
                                                       #   pattern) that calls the reconnect.go export,
                                                       #   passed to xcecli_set_reconnect_handler; the
                                                       #   heartbeat tick callback explicitly re-invokes
                                                       #   the same rejoin path on recovery (see below)
src/frontend/apiserver/backend/health.go              # NEW: backend-degraded circuit-breaker state
src/frontend/apiserver/backend/reconnect.go           # NEW: exported Go rejoin logic (re-sends
                                                       #   xcecli_join for HostRole_apiserver, clears
                                                       #   health.go state); called from clib.go's C
                                                       #   shim AND directly from the heartbeat callback
src/frontend/apiserver/backend/heartbeat.go           # MODIFIED: tick callback detects a
                                                       #   degraded-to-healthy transition after
                                                       #   xcecli_heartbeat succeeds and explicitly
                                                       #   triggers reconnect.go's rejoin logic
src/frontend/apiserver/utils/config.go                # no schema change (Peers already exists)

src/worker/testsuite/test_e2e_multimaster_join.c       # NEW: candidate-list join + reconnect E2E
src/scheduler/testsuite/test_e2e_multimaster_join.c    # NEW: same, scheduler
src/frontend/apiserver/backend/health_test.go          # NEW: circuit-breaker unit tests
src/frontend/apiserver/backend/clib_test.go            # + candidate-list parity tests (existing file)

controller/ and libxce's failover mechanism itself are not modified — only consumed.

System Flows

Startup connect (Req 1)

sequenceDiagram
    participant Daemon
    participant Convert as xce_cluster_config_to_client_config
    participant Ctx as xcecli_ctx (client.c)
    participant CandA as Candidate A (standby)
    participant CandB as Candidate B (primary)

    Daemon->>Convert: full candidate list (validated)
    Convert-->>Daemon: xce_client_config_t (nmasters=N)
    Daemon->>Ctx: xcecli_ctx_init + xcecli_start
    Ctx->>CandA: connect
    CandA-->>Ctx: connected, not primary
    Ctx->>Daemon: xcecli_join (role)
    Ctx->>CandA: join request
    CandA-->>Ctx: standby redirect
    Ctx->>Ctx: rotate to next candidate
    Ctx->>CandB: connect + join
    CandB-->>Ctx: joined as primary
    Ctx-->>Daemon: xcecli_join success

Runtime failover (Req 2, Req 3)

sequenceDiagram
    participant Daemon
    participant Ctx as xcecli_ctx lock held
    participant OldPrimary as Old Primary dies
    participant NewPrimary as New Primary

    Daemon->>Ctx: xcecli_* call existing traffic
    Ctx->>OldPrimary: send
    OldPrimary--xCtx: connection lost
    Ctx->>Ctx: XCECLI_FAILOVER_CALL rotates and backs off
    Ctx->>NewPrimary: reconnect
    NewPrimary-->>Ctx: connected
    Ctx->>Daemon: invoke rejoin callback lock still held
    Note over Daemon: callback only sets a needs-rejoin flag, no I/O, returns 0
    Ctx->>NewPrimary: resume original call
    NewPrimary-->>Ctx: reply
    Ctx-->>Daemon: original call result, lock released
    Note over Daemon: after the call returns, lock is free
    Daemon->>Daemon: check and clear needs-rejoin flag
    Daemon->>Ctx: xcecli_join role separate call
    Ctx->>NewPrimary: join
    NewPrimary-->>Ctx: joined
    Ctx-->>Daemon: rejoin complete

Key decision reflected in the diagram: the rejoin callback invoked by the library never performs I/O — it only flags that a rejoin is needed (see "Rejoin callback constraint" above, avoiding a self-deadlock on the still-held per-ctx lock). The actual xcecli_join call happens as a distinct follow-up call from the daemon, on the same thread (worker: connector thread; scheduler: allocator thread; apiserver: the requesting goroutine), once the triggering call has returned and released the lock — no separate reconnect loop or new thread is introduced either way.

apiserver fail-fast while degraded (Req 4)

sequenceDiagram
    participant Client as API Client
    participant Handler as apiserver handler
    participant Health as health.go
    participant Ctx as xcecli_ctx (gCtx)

    Client->>Handler: HTTP request needing backend
    Handler->>Health: is backend degraded?
    alt degraded
        Health-->>Handler: yes
        Handler-->>Client: 503 service unavailable (fast)
    else healthy
        Health-->>Handler: no
        Handler->>Ctx: backend call
        alt call exhausts retry budget
            Ctx-->>Handler: error
            Handler->>Health: mark degraded (with TTL)
            Handler-->>Client: 503 service unavailable
        else call succeeds
            Ctx-->>Handler: result
            Handler-->>Client: 200 with result
        end
    end

apiserver idle-period reconnect (Req 2.4)

sequenceDiagram
    participant HB as heartbeat.go tick
    participant Ctx as xcecli_ctx (gCtx, FF_CALL)
    participant Rejoin as reconnect.go
    participant Health as health.go
    participant NewPrimary as New Primary

    Note over HB: no client requests in flight
    HB->>Ctx: xcecli_heartbeat (fire-and-forget)
    Ctx->>Ctx: connection known-dead: best-effort rotate + reconnect
    Ctx->>NewPrimary: reconnected transport only
    Ctx-->>HB: rc success (was failing, now succeeds)
    HB->>HB: detect failure-to-success transition
    HB->>Rejoin: explicitly trigger rejoin
    Rejoin->>NewPrimary: xcecli_join(HostRole_apiserver)
    NewPrimary-->>Rejoin: joined
    Rejoin->>Health: MarkHealthy()

This path exists because XCECLI_FF_CALL (fire-and-forget, used by the heartbeat) never invokes the library's own rejoin hook — only the full-retry path (XCECLI_FAILOVER_CALL, used by write calls) does. Without this explicit trigger, an idle apiserver instance would repair its TCP connection on every heartbeat tick but never re-establish cluster membership, so push notifications would stay dead until an unrelated write request happened to arrive.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1-1.3	Try each candidate in order at startup	libxce (unchanged), ConfigConverter	xcecli_start, xcecli_join	Startup connect
1.4	Worker uses full list on FIRST attempt	WorkerControllerConnector	controller_connector.c init order fix	Startup connect
1.5	Single/no-candidate mode unchanged	All three	ConfigConverter (nmasters=1 path)	—
1.6	Clear startup failure if all candidates fail	All three	existing xcecli_start/xcecli_join return codes	Startup connect
2.1-2.3	Automatic reconnect + retry until success or shutdown	libxce (unchanged), all three's rejoin callbacks	xcecli_set_reconnect_handler	Runtime failover
2.4	apiserver resumes push notifications after reconnect (incl. idle-period recovery)	ApiserverBackendBridge	reconnect.go's actual-rejoin function re-sends xcecli_join(HostRole_apiserver), triggered via the shared post-call helper's flag check on write-path reconnect AND explicitly from heartbeat.go on idle-period recovery	Runtime failover, apiserver idle-period reconnect
3.1-3.2	Retry/backoff doesn't block other liveness work	WorkerControllerConnector (thread isolation preserved), SchedulerControllerClient (allocator/picker thread separation)	existing thread structure	Runtime failover
4.1-4.2	apiserver fails fast to clients while degraded	ApiserverBackendHealth	health.go	apiserver fail-fast flow
5.1-5.2	Per-candidate connect/reconnect logging	All three	log statements at each rotation	Startup connect, Runtime failover
6.1-6.2	Consistent candidate-list syntax	ApiserverConfigParity	CGO call to xce_parse_peers/xce_cluster_config_validate	Startup connect

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies	Contracts
ConfigConverter	libxce (C)	Convert validated candidate list to client config	1.1-1.6, 6.1	xce_cluster_config_validate (P0)	Service
WorkerControllerConnector	worker (C)	Single init/connect/join + rejoin for worker	1.1-1.4, 2.1-2.3, 3.1-3.2, 5.1-5.2	ConfigConverter (P0), xcecli_* (P0)	Service
SchedulerControllerClient	scheduler (C)	Single init/connect/join + rejoin for scheduler	1.1-1.3, 1.5-1.6, 2.1-2.3, 3.1-3.2, 5.1-5.2	ConfigConverter (P0), xcecli_* (P0)	Service
ApiserverBackendBridge	apiserver (Go/CGO)	Single init/connect/join for apiserver; config parity	1.1-1.6, 2.4, 5.1-5.2, 6.1-6.2	ConfigConverter via CGO (P0), xcecli_* via CGO (P0)	Service
ApiserverBackendHealth	apiserver (Go)	Fail-fast circuit breaker for backend-down state	4.1-4.2	ApiserverBackendBridge (P0, consumes rejoin/error signals)	State

Shared Library

ConfigConverter

Field	Detail
Intent	Turn a validated xce_cluster_config_t into a fully-populated xce_client_config_t, clamped to N_MAX_MASTERS with a WARN on truncation
Requirements	1.1, 1.2, 1.3, 1.5, 1.6, 6.1

Responsibilities & Constraints

• Pure, socket-free transformation (mirrors xce_test's existing inline logic at test.c:360-372, generalized into one reusable function).
• Does not call xce_cluster_config_validate() itself — callers validate first (existing contract, unchanged) and pass a pre-validated config.
• Does not own from/key (identity/auth) — caller sets those on the output struct after conversion, matching how xce_test, worker, and scheduler already source them independently (CLI/env-specific).

Dependencies

• Inbound: WorkerControllerConnector, SchedulerControllerClient, ApiserverBackendBridge (all P0)
• Outbound: none (leaf function)

Contracts: Service [x]

Service Interface

/* xce/cluster.h */
/**
 * xce_cluster_config_to_client_config() — convert a validated candidate list
 * into an xce_client_config_t's master fields (nmasters/masters/master_ports).
 *
 * Copies up to N_MAX_MASTERS candidates from cfg->candidates[] in order.
 * If cfg->candidates_count > N_MAX_MASTERS, copies the first N_MAX_MASTERS and
 * logs a WARN identifying the truncated count (mirrors xce_test's existing
 * clamp behavior).
 *
 * Does NOT set out->from / out->key — caller sets those separately.
 * Does NOT validate cfg — caller must have already called
 * xce_cluster_config_validate() successfully.
 *
 * Returns 0 on success, -EINVAL if cfg or out is NULL or cfg->candidates_count < 1.
 */
int xce_cluster_config_to_client_config(const xce_cluster_config_t* cfg,
                                        xce_client_config_t* out);

• Preconditions: cfg already passed xce_cluster_config_validate().
• Postconditions: out->nmasters in [1, N_MAX_MASTERS]; out->masters[i]/out->master_ports[i] populated for i < out->nmasters, in the same order as cfg->candidates[].
• Invariants: never writes beyond N_MAX_MASTERS entries regardless of cfg->candidates_count.

Worker

WorkerControllerConnector

Field	Detail
Intent	Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect
Requirements	1.1-1.4, 2.1-2.3, 3.1-3.2, 5.1-5.2

Responsibilities & Constraints

• Replaces xcecd_join_cluster's per-candidate loop (controller_connector.c:237-311) with: build xce_client_config_t via ConfigConverter once (candidates from cc_ctx->cluster_cfg when --peers given, else today's single-host fallback path unchanged per Req 1.5), then one xcecli_ctx_init + xcecli_start + xcecli_join.
• Fixes the init-order bug by moving cc_ctx->cluster_cfg population (main.c) to before xcecd_start() is called, so the very first connect already sees the full candidate list (Req 1.4).
• Registers a rejoin callback (xcecli_set_reconnect_handler) that only sets a needs_rejoin flag and returns 0 — per "Rejoin callback constraint" above, it must not call xcecli_join (or any xcecli_* function) directly, since the library invokes it while ac->failover.lock is still held.
• checking_connection()'s periodic xcecli_hello keeps its existing role (liveness probe on the connector thread) but no longer needs to manually call xcecd_join_cluster on failure — a transport failure inside xcecli_hello already triggers the library's transparent reconnect + the registered flag-setting callback. Immediately after each xcecli_hello call returns (lock now free), checking_connection() checks and clears needs_rejoin; if set, it calls xcecli_join(ctx, HostRole_worker, &host_attrs) as a separate call — the same join payload used at startup.
• Runs entirely on the existing dedicated connector thread (start_controller_connector_thread/connector_to_controller), preserving isolation from hardware_monitor/system_monitor/process_manager update paths (Req 3.1) — no new thread introduced.

Dependencies

• Inbound: xcecd_start() (P0)
• Outbound: ConfigConverter (P0), xcecli_* (P0)

Contracts: Service [x]

Implementation Notes

• Integration: preserve existing WARN-level log statements around standby-redirect and candidate-exhaustion, now emitted from the library's classification path or immediately around the single init/join call, so Req 5 diagnostics are not lost in the simplification.
• Validation: implementation must confirm (code read, not test-only) that hardware_monitor_update/system_monitor_update/process_manager_update do not execute on the connector thread — flagged as a Validation Hook, not re-verified in this design pass.
• Risks: deleting the hand-rolled loop must not silently drop the MAX_RETRY_COUNT/RETRY_INTERVAL bounded-retry-then-handle_disconnect() behavior in checking_connection() — that outer bound (Req 2.3's "keep retrying" vs. any existing bounded-give-up behavior) must be reconciled explicitly during implementation, not left ambiguous.

Scheduler

SchedulerControllerClient

Field	Detail
Intent	Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect
Requirements	1.1-1.3, 1.5-1.6, 2.1-2.3, 3.1-3.2, 5.1-5.2

Responsibilities & Constraints

• Replaces sched_init's inline per-candidate loop (scheduler.c:462-495) with ConfigConverter + one xcecli_ctx_init/xcecli_start/xcecli_join call.
• Registers a rejoin callback that only sets a needs_rejoin flag and returns 0 (same constraint as worker — the library invokes it while ac->failover.lock is held, so it must not call xcecli_join itself). After each xcecli_jobdispatch call returns, the allocator checks and clears the flag; if set, it calls xcecli_join(ctx, HostRole_scheduler, NULL) as a separate call before continuing.
• No new thread: xcecli_jobdispatch (the only backend call site, scheduler.c:1289) continues to run from the existing sched_allocator thread; blocking during a reconnect/backoff cycle is an accepted, benign degradation since the allocator has no useful work without a controller anyway, and sched_job_picker runs on its own separate thread unaffected (Req 3.1-3.2 satisfied by the existing thread split, not new code).

Dependencies

• Inbound: sched_init() (P0)
• Outbound: ConfigConverter (P0), xcecli_* (P0)

Contracts: Service [x]

Implementation Notes

• Integration: preserve the existing WARN/INFO log lines around candidate parsing (main.c) and add per-attempt/per-outcome logging around the single init/join call (today's loop logged each attempt; the library's own classification path should be checked for equivalent visibility, else add it locally).
• Risks: today's total absence of reconnect logic means this is additive behavior change (scheduler previously required a manual restart after primary loss) — call out explicitly in the task/rollout notes so operators aren't surprised by scheduler now retrying indefinitely instead of exiting.

apiserver

ApiserverBackendBridge

Field	Detail
Intent	Single connect/join at startup using the full candidate list via the C library (through CGO), replacing the Go-level per-peer loop; config-syntax parity with scheduler/worker
Requirements	1.1-1.6, 2.4, 5.1-5.2, 6.1-6.2

Responsibilities & Constraints

• RegisterToBackend() (clib.go) is rewritten: build a C xce_cluster_config_t from config.Peers (or the single Address/Port fallback, Req 1.5) by calling xce_parse_peers via CGO (not the existing hand-written Go parsePeer), validate with xce_cluster_config_validate, convert with ConfigConverter, then call xce_backend_init once with the resulting nmasters > 1 config — replacing the current for _, peer := range config.Peers { tryConnect(...) } Go loop entirely.
• reconnect.go holds two plain Go functions: (1) a //export-annotated flag-setter — registered as the library's rejoin callback, does no I/O, only sets a needs_rejoin atomic flag and returns 0 (per "Rejoin callback constraint" above: the library invokes it while ac->failover.lock is held, so it must not call xcecli_join/any CGO backend function itself); (2) a plain Go function that performs the actual rejoin — calls xcecli_join(gCtx.ctx, HostRole_apiserver, NULL) and clears ApiserverBackendHealth's degraded state on success.
• The flag-setter is registered with the C library from clib.go's own cgo preamble, via a small static C shim (xce_backend_rejoin_handler, mirroring the existing xce_backend_handler/go_push_handler pattern already used for push dispatch) passed to xcecli_set_reconnect_handler — cgo symbol resolution is per-file (see wire_dispatch_support.go's documented rationale for the identical go_push_handler wrapper), so the shim and the registration call must live in the same file as the extern declaration, not standalone in reconnect.go.
• Rather than duplicating the same pre-call/post-call logic across all four wrapper functions, the pre-call IsDegraded() fail-fast check (ApiserverBackendHealth, below) and the post-call needs_rejoin check-and-clear are both implemented once, in a single shared internal helper each of the four clib.go wrappers calls around its C.xcecli_* invocation — not copy-pasted per wrapper.
• Idle-period reconnect gap (Req 2.4) fix: XCECLI_FF_CALL — the macro backing xcecli_heartbeat, heartbeat.go's existing periodic prober (cmd/main.go:76) — performs a best-effort transport reconnect on a known-dead connection but never invokes the library's rejoin hook (verified against client_internal.h/client_failover.c: xcecli_invoke_rejoin is called only from xcecli_retry_next, the full-retry path used by write calls, never from the fire-and-forget path). Left as-is, a controller failover occurring while apiserver has no in-flight write request would silently repair the TCP connection without ever re-joining, permanently starving push notifications until an unrelated write request happens to arrive. Fix: the heartbeat tick callback (heartbeat.go) tracks the previous tick's outcome; when a tick transitions from failure to success (transport just recovered), it explicitly calls reconnect.go's rejoin function itself, then clears ApiserverBackendHealth. This reuses the same rejoin logic Task 4.3 already builds — no duplicate join-payload code, just an explicit trigger point the write-call path already gets for free from the library.

Dependencies

• Inbound: cmd/main.go (P0)
• Outbound: ConfigConverter via CGO (P0), xcecli_* via CGO (P0), ApiserverBackendHealth (P1, signals it on outcome)

Contracts: Service [x]

Service Interface

// backend/clib.go
// RegisterToBackend initialises the client using the full configured
// candidate list (config.Peers, or the single Address/Port fallback) and
// registers the rejoin callback. Single call — no per-peer Go loop.
func RegisterToBackend(config *BackendConfig) error

• Preconditions: StartNotificationListener already called (unchanged from today).
• Postconditions: on success, gCtx.ctx is joined with nmasters reflecting every configured candidate (clamped per ConfigConverter); the rejoin callback is registered.
• Invariants: candidate-list parsing/validation errors surface the same way xce_parse_peers/xce_cluster_config_validate report them in C (Req 6.2) — not a separately-worded Go error.

Implementation Notes

• Integration: parsePeer (Go) and its tests are removed/retired in favor of the CGO-routed parser (Req 6.1).
• Integration: the rejoin C shim must be defined in clib.go's cgo preamble (not reconnect.go) — see Responsibilities above; this was the design's one build-risk item, now resolved by construction rather than left as an implementation-time surprise.
• Risks: CGO error surfaces (C -EINVAL etc.) need a clear Go-side error message mapping so operators get an actionable message (Req 1.6, 6.2), not a bare error code.

ApiserverBackendHealth

Field	Detail
Intent	Let request handlers fail fast (503) instead of blocking on the library's full retry budget when the backend is known-degraded
Requirements	4.1, 4.2

Responsibilities & Constraints

• New health.go: an atomic "degraded-until" timestamp. IsDegraded() bool — true if now < degraded-until. MarkDegraded(ttl time.Duration) — called by backend-call wrapper functions in clib.go when a call's returned error indicates retry-budget exhaustion (e.g. E_XCE_NOT_READY). MarkHealthy() — called by the rejoin callback (reconnect.go) on a successful rejoin.
• Backend-call wrapper functions used by request-handling code paths (job submit, job control, queue control, host control, etc. — the existing gCtx.ctx-based calls in clib.go) check IsDegraded() first, via the shared internal helper described under ApiserverBackendBridge above (not duplicated per wrapper); if true, return a clear "backend unavailable" error immediately instead of invoking the blocking C call (Req 4.1). Calls that don't require the backend are unaffected (Req 4.2 — nothing to change there, they already don't touch gCtx).

Dependencies

• Inbound: ApiserverBackendBridge (sets state on outcomes)
• Outbound: none

Contracts: State [x]

State Management

• State model: single atomic degraded-until timestamp (int64 unix-nano via sync/atomic, same pattern as existing heartbeatIntervalSecs in heartbeat.go).
• Persistence & consistency: process-local, in-memory only; reset on apiserver restart (matches the rest of apiserver's in-memory state model per tech.md).
• Concurrency strategy: lock-free atomic read/write; safe for concurrent goroutines per request.

Implementation Notes

• Integration: TTL-based clearing (Risks & Mitigations in research.md) in addition to the rejoin-callback clear, so a missed/edge-case rejoin doesn't permanently wedge the flag.
• Validation: unit tests cover degraded→healthy transitions via both the TTL expiry and explicit MarkHealthy().

Error Handling

Error Strategy

No new error taxonomy — reuse the existing controller status codes (E_XCE_REDIRECT/E_XCE_STANDBY/E_XCE_NOT_READY, errnox.h) that client_failover.c already classifies, and the existing xce_cluster_config_validate() -EINVAL cases for malformed candidate-list config.

Error Categories and Responses

• Malformed candidate-list config (Req 1.6, 6.2): xce_parse_peers/xce_cluster_config_validate failure at startup → each component reports a clear message naming the offending entry and refuses to start (unchanged behavior for worker/scheduler; apiserver gains the same via the CGO route).
• All candidates unreachable at startup (Req 1.6): xcecli_start/xcecli_join returns the last transport error after exhausting all candidates → component reports which candidates were attempted (Req 5.2) and fails startup (worker/scheduler: process exits as today; apiserver: RegisterToBackend returns an error, cmd/main.go's existing handling applies unchanged).
• Runtime disconnect (Req 2): transparently retried/rotated by the library; components only observe it through their own periodic calls' return codes and the rejoin callback — no new error category, just new logging around the existing outcomes.
• Candidate count exceeds N_MAX_MASTERS: ConfigConverter clamps and WARNs (not an error) — matches xce_test's existing behavior exactly.
• apiserver backend degraded (Req 4.1): mapped to a clear 503-style API error, distinct from other 5xx causes, so operators/clients can distinguish "backend down, retrying" from an unrelated internal error.

Monitoring

Per-candidate connect/reconnect attempts and outcomes are logged (Req 5) using each component's existing logger (XCECD_LOG for worker, the scheduler's existing INFO/WARN macros, zap for apiserver) — no new logging infrastructure, just new call sites around the simplified connect/reconnect path.

Testing Strategy

• Unit Tests:
  • ConfigConverter: clamps at N_MAX_MASTERS with WARN when given more candidates; preserves candidate order; rejects NULL/empty input (test_cluster.c).
  • apiserver health.go: degraded→healthy via MarkHealthy(), via TTL expiry; concurrent goroutine safety (health_test.go).
  • apiserver CGO-routed parser: same malformed-input cases scheduler/worker already reject, now asserted through the Go call site (clib_test.go).
  • Rejoin flag-setter callbacks (worker, scheduler, apiserver): with a real ac->failover.lock held (simulating the library's call context per "Rejoin callback constraint"), invoking the registered callback returns promptly without blocking and performs no I/O — a regression guard against reintroducing the self-deadlock this design pass found.
• Integration Tests:
  • Worker/scheduler startup against a 3-candidate config where candidate[0] is a live standby — assert the component joins as expected without operator intervention (Req 1.1-1.3), extending each component's testsuite/.
  • Worker/scheduler reconnect: kill the currently-joined primary mid-session, assert the component reconnects and re-joins the new primary without restart (Req 2.1-2.3), reusing the project's existing /xce-deploy-ha + /xce-test-ha pattern already used for controller HA soak testing — this is also the test that would hang (not fail cleanly) if the self-deadlock were reintroduced, so it must run with a wall-clock timeout that fails loud rather than hanging CI.
• E2E Tests:
  • Full 3-node HA cluster with worker + scheduler + apiserver all configured with the full candidate list; kill the primary; assert all three reconnect, worker resumes job execution reporting, scheduler resumes dispatch, apiserver resumes serving requests and receiving push notifications (Req 2.4) — new test_e2e_multimaster_join.c per C component plus an apiserver-side check in the existing HA soak harness.
  • apiserver idle-period reconnect: kill the primary while apiserver has no in-flight write request (only the heartbeat prober running); assert apiserver still rejoins and resumes push notifications, proving the heartbeat-triggered rejoin path (not just the write-call-triggered one) works (Req 2.4).
  • apiserver fail-fast: with the backend killed and not yet reconnected, assert an API request returns quickly (bounded, sub-second) with a clear 503 rather than hanging for the full retry budget (Req 4.1).

Security Considerations

No new trust boundary: candidate hosts are the same set of controllers already trusted today; TLS/identity handling is unchanged (see Non-Goals — CA-cert wiring is out of scope and pre-existing). No new authentication surface is introduced.