myxce/.kiro/specs/component-failover-adoption/design.md

Design Document: component-failover-adoption

Overview

Scheduler, worker, and apiserver each connect to the controller cluster today using only a single configured endpoint at a time, with divergent, partly-broken hand-rolled logic for trying alternates. Meanwhile libxce already ships a complete, tested, transparent multi-master mechanism (XCECLI_FAILOVER_CALL and friends in src/lib/client.c) — the same one xce_test's masters setting already uses correctly. This design wires all three components onto that existing mechanism: populate the full candidate list once, let the library rotate/retry/reconnect on every call, and register a rejoin callback so a component re-establishes cluster membership after a transparent reconnect.

Users: HA cluster operators who configure multiple controller candidates (--peers for scheduler/worker, backend.peers YAML for apiserver) and expect scheduler, worker, and apiserver to survive a controller failover without manual restarts — the same expectation xce_test already meets.

Impact: Removes worker's and scheduler's hand-rolled per-candidate join loops (deleting, not adding, most of the connection-retry code in controller_connector.c and scheduler.c); replaces apiserver's Go-level per-peer loop with a single library-driven connect; adds one new small shared C helper and a small Go-side health guard. No changes to the controller or to libxce's failover mechanism itself.

Goals

• Scheduler, worker, and apiserver each hand the full configured candidate list to xce_client_config_t on the very first connection attempt (Req 1).
• All three automatically reconnect and resume cluster membership after their controller connection is lost, without a process restart (Req 2).
• Worker's and scheduler's retry/backoff never blocks duties independent of controller connectivity (Req 3).
• apiserver returns a clear, fast error to its own API clients while its backend connection is down, instead of hanging (Req 4).
• Every connect/reconnect attempt and outcome is logged per-candidate (Req 5).
• apiserver accepts the identical candidate-list syntax scheduler/worker already accept (Req 6).
• Scheduler decision publishing (xcecli_jobdispatch/xcecli_set_pend_reason) never silently drops a decision on retry-budget exhaustion — it keeps retrying that decision until success or shutdown, so scheduler-local and controller-observed job state cannot permanently diverge (Req 7).

Non-Goals

• Changing xce_test (it is already the reference-correct client) — its clamp already references N_MAX_MASTERS symbolically (test.c:372-378), so it auto-absorbs the constant reconciliation below with zero code change.
• Completing or adopting the separate, unused xce_connect_cluster()/xce_reconnect_* primitive in cluster.c — superseded by the mechanism this design adopts; left untouched.
• Wiring CA-cert/TLS trust into the real connect path — xce_client_config_t carries no such field today and this feature does not add one; TLS is established the same way it already is.
• Controller-side changes (redirect-reply population, peer authentication) — already shipped by libxce-failover.
• Recovering worker job-state lost during a disconnect window — owned by worker-rejoin-resync.
• Persisting queued-but-unpublished scheduler decisions across a scheduler process crash/restart (Req 7 AC6) — Req 7's fix keeps a running scheduler process from giving up early on a network retry; it adds no WAL/persistence for the in-memory decisions/jbuf queues, so a scheduler crash mid-outage still loses whatever was queued, exactly as it does today.

Boundary Commitments

This Spec Owns

• A new shared libxce helper that converts a validated xce_cluster_config_t (parsed candidate list) into a fully-populated xce_client_config_t.
• Worker's, scheduler's, and apiserver's integration code that calls xcecli_ctx_init/xcecli_start/xcecli_join with the full candidate list and registers a rejoin callback.
• apiserver's candidate-list config parsing (routed through the existing C parser via CGO instead of a second Go implementation).
• apiserver's client-facing fail-fast behavior while its backend connection is degraded.
• Connect/reconnect diagnostic logging in all three components.
• Scheduler's decision-publish retry-until-success-or-shutdown wrapper around xcecli_jobdispatch/xcecli_set_pend_reason (Req 7) — closes the silent-drop gap found in the 2026-08-10 design validation.
• Reconciling N_MAX_MASTERS (client.h) to derive from MAX_HA_CANDIDATES (config.h) (Req 1 AC7, scope expanded 2026-08-10 at explicit user direction — supersedes this design's earlier Non-Goal treating the mismatch as pre-existing/out-of-scope): config.h becomes the single source of truth for "max configured controller candidates." Confirmed safe by source audit — no wire-protocol array assumes 3 (masters[] is never serialized; the controller's own peer list is a dynamic capnp List(CtrlPeer)), and config.h has no include-cycle risk (it depends only on <stdint.h>; client.h already transitively sees MAX_HA_CANDIDATES via xceapi.h/xce/types.h → config.h).
• Eliminating the legacy single-host code path in worker, scheduler, and apiserver in favor of a single unified pipeline (Req 1 AC5, redefined 2026-08-10): today all three synthesize a nmasters=1 xce_client_config_t directly from legacy fields, bypassing xce_cluster_config_t/xce_parse_peers/xce_cluster_config_validate/ConfigConverter entirely (controller_connector.c:129-160's init_client_config(), scheduler.c:613's single-host else branch, clib.go:79-101's tryConnect() direct-CGO-struct path). This spec now closes that parallel path: a single controller is the candidates_count=1 case of the exact same validated pipeline, not a separately-maintained mechanism.

Out of Boundary

• xce_test and its masters setting (unchanged, reference implementation).
• libxce's failover mechanism (client.c's connect/rotate/retry/backoff logic, client_internal.h, client_failover.c) — consumed as-is, not modified. Distinct from client.h's N_MAX_MASTERS constant, which this spec does change (above) — the mechanism and the struct-layout constant are different surfaces with different boundaries.
• xce_connect_cluster()/xce_reconnect_* (unused, untouched).
• Controller-side membership/redirect logic.
• Worker job-state resync content on rejoin (worker-rejoin-resync).
• TLS/CA-cert plumbing into the real connect path.

Allowed Dependencies

• src/lib/client.c / client_internal.h / client_failover.h (shipped libxce-failover mechanism) — read-only dependency, no modification.
• src/lib/cluster.c / xce/cluster.h (xce_parse_peers, xce_cluster_config_validate) — existing, unmodified; this spec adds one new function alongside them.
• Existing per-component candidate-list CLI/YAML surfaces (--peers, --ca-cert, backend.peers) — unmodified externally; only how they're consumed changes.

Revalidation Triggers

• Any change to xce_client_config_t's layout, or any future change to N_MAX_MASTERS/MAX_HA_CANDIDATES that reintroduces divergence between them (this spec establishes them as one value — a regression that redefines N_MAX_MASTERS independently of config.h again should re-trigger review).
• Any change to XCECLI_FAILOVER_CALL/XCECLI_FF_CALL retry/classification semantics (libxce-failover).
• Any change to xce_parse_peers/xce_cluster_config_validate syntax or validation rules (apiserver's CGO-based parser would need re-verification).
• worker-rejoin-resync landing — worker's rejoin callback becomes the natural place to also trigger job-state resync; this spec's callback is a no-op-beyond-rejoin placeholder for that.

Architecture

Existing Architecture Analysis

Today each of the three components independently drives its own xcecli_ctx_t lifecycle with nmasters hardcoded to 1 per connection attempt, and layers its own outer candidate loop on top:

• worker: controller_connector.c's xcecd_join_cluster loops candidates itself, but the very first connection (inside xcecd_start()) happens before cc_ctx->cluster_cfg is even populated (main.c assigns it after xcecd_start() returns) — the candidate list is silently ignored on startup.
• scheduler: scheduler.c's sched_init loops candidates correctly at startup, but there is no reconnect logic anywhere afterward.
• apiserver: clib.go's RegisterToBackend loops candidates once via a hand-written Go parser, only at process startup.

None of the three ever set nmasters > 1 on a single xce_client_config_t, so the library's own rotation/retry (client.c, wrapped by XCECLI_FAILOVER_CALL) never actually engages for them.

Architecture Pattern & Boundary Map

graph TB
    subgraph Daemons
        Worker[worker xcecd]
        Scheduler[xce_scheduler]
        Apiserver[apiserver backend package]
    end

    subgraph NewSharedCode
        Convert[xce_cluster_config_to_client_config]
    end

    subgraph ExistingLibxce
        Parse[xce_parse_peers]
        Validate[xce_cluster_config_validate]
        CtxInit[xcecli_ctx_init xcecli_start xcecli_join]
        Failover[XCECLI_FAILOVER_CALL rotate retry backoff]
        Rejoin[rejoin callback xcecli_set_reconnect_handler]
    end

    subgraph Cluster
        Primary[Controller Primary]
        Standby1[Controller Standby]
    end

    Worker --> Parse
    Scheduler --> Parse
    Apiserver --> Parse
    Parse --> Validate
    Validate --> Convert
    Convert --> CtxInit
    CtxInit --> Failover
    Failover --> Rejoin
    Failover --> Primary
    Failover --> Standby1

Architecture Integration:

• Selected pattern: thin per-component integration layer over an unchanged, already-shipped library mechanism (adapter, not new subsystem).
• Domain boundaries: libxce owns candidate parsing, validation, connect/rotate/retry/backoff, and rejoin-hook invocation (unchanged). Each daemon owns only: building the candidate list from its own config surface, calling the library once, and supplying its role-specific rejoin behavior (what to send when re-joining).
• Existing patterns preserved: the xce_parse_peers/xce_cluster_config_validate/xce_client_config_t split; the CGO bridge pattern already used throughout backend/; worker's dedicated connector thread.
• New components: one C helper (candidate-list → client-config conversion, generalized out of xce_test's inline logic) and one small Go health-state module (apiserver-only, has no C-side equivalent need).
• Steering compliance: no new blocking work introduced on iomgr/delaymgr-equivalent threads; worker's existing thread isolation is preserved, not altered.

Rejoin callback constraint: avoiding self-deadlock on ac->failover.lock

Verified directly against client_failover.c:341-411: XCECLI_FAILOVER_CALL holds ac->failover.lock (a plain, non-recursive pthread_mutex_t — mylock_init calls pthread_mutex_init(l, NULL), no PTHREAD_MUTEX_RECURSIVE attribute) for the entire retry loop, and xcecli_invoke_rejoin() — which calls the registered rejoin callback — is invoked synchronously from inside that loop, while the lock is held (client_failover.c:397). Any rejoin callback that itself calls a public xcecli_* wrapper on the same xcecli_ctx_t (e.g. xcecli_join, which is itself XCECLI_FAILOVER_CALL-wrapped and tries to re-lock the same mutex) self-deadlocks — the calling thread blocks forever on its own already-held lock. This was never caught because xcecli_set_reconnect_handler has zero production callers today and the existing unit test's callback is a no-op probe that never calls back into the client.

This directly constrains every component's rejoin implementation in this spec:

• The callback registered with the library (worker's checking_connection→xcecli_hello path, scheduler's xcecli_jobdispatch path, apiserver's write-call path) must do no I/O and call no xcecli_* function — it may only set a lightweight "needs rejoin" flag (a plain bool/atomic on the consumer side) and return 0 so the interrupted operation resumes normally.
• Each component's own calling code, after the triggering xcecli_* call returns (at which point the library has released the lock), checks that flag and, if set, issues xcecli_join(...) as a separate, subsequent call — safe, since the lock is free by then. The flag is cleared only once that xcecli_join call itself succeeds, never merely because it was attempted (Req 2 AC5, gap found in the 2026-08-10 design validation's second pass): xcecli_join is itself XCECLI_FAILOVER_CALL-wrapped and can exhaust its own retry budget and return -E_XCE_NOT_READY just like any other wrapped call. Clearing the flag unconditionally before or regardless of that outcome would leave the component transport-reconnected but not cluster-joined, with nothing left to retry the join until the next full disconnect/reconnect cycle happens to fire the rejoin callback again — an indefinite, silent orphaning, not a bounded delay. On failure the flag stays set, so the component's next naturally-occurring opportunity (worker: next xcecli_hello return; scheduler: next decision-publish return, itself now retried indefinitely per Req 7; apiserver: next backend-call wrapper return) re-attempts the join.
• apiserver's heartbeat-triggered path (see below) is unaffected by this constraint — it already detects recovery and calls the rejoin logic after xcecli_heartbeat has returned (XCECLI_FF_CALL also releases its lock before returning), so it was never at risk.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Shared library	C (gnu17), src/lib/cluster.c	New candidate→client-config conversion helper	Extends existing file alongside xce_parse_peers
Worker / Scheduler	C (gnu17)	Consume the existing xcecli_ctx_t mechanism directly	No new dependency
apiserver	Go 1.23.4 + CGO	Consume the same mechanism via the existing CGO bridge; new Go-side health-state guard	sync/atomic, consistent with existing heartbeat.go pattern

File Structure Plan

Modified / New Files

src/include/client.h                                  # fix: N_MAX_MASTERS derives from
                                                       #   config.h's MAX_HA_CANDIDATES (3->4)
                                                       #   instead of an independent literal (Req 1.7)
src/include/xce/cluster.h                          # + declare xce_cluster_config_to_client_config()
src/lib/cluster.c                                    # + implement xce_cluster_config_to_client_config()
src/lib/testsuite/test_cluster.c                      # + unit tests for the new helper
src/lib/testsuite/test_libxce_retry.c                 # fix: fo_make_client_masters() currently
                                                       #   hardcodes 3 literal host/port entries then
                                                       #   copies N_MAX_MASTERS slots -- update to match
                                                       #   the raised N_MAX_MASTERS, not silently copy
                                                       #   an uninitialized 4th slot

src/worker/src/controller_connector.c                # rewrite: delete init_client_config()'s
                                                       #   standalone xce_client_config_t construction
                                                       #   and xcecd_join_cluster()'s separate
                                                       #   single-host-fallback branch (both bypassed
                                                       #   xce_cluster_config_t entirely) -- single
                                                       #   init+start+join via the one synthesize-
                                                       #   validate-convert-connect sequence, whether
                                                       #   the candidate list came from --peers or was
                                                       #   synthesized as one candidate (Req 1.5);
                                                       #   register rejoin callback
src/worker/src/main.c                                 # fix: populate cc_ctx->cluster_cfg BEFORE
                                                       #   xcecd_start() (removes init-order bug);
                                                       #   fix hardcoded `if (candidates_count > 3)`
                                                       #   (main.c:414) to reference N_MAX_MASTERS
                                                       #   symbolically instead of a literal 3, so it
                                                       #   tracks the raised value automatically
src/worker/include/controller_connector.h             # + rejoin callback function declaration

src/scheduler/src/scheduler.c                         # rewrite: sched_init uses the new helper +
                                                       #   single init+start+join; delete inline
                                                       #   per-candidate loop AND the separate
                                                       #   "single-host path (original)" else branch
                                                       #   (scheduler.c:613, bypassed
                                                       #   xce_cluster_config_t entirely) -- single-host
                                                       #   is now candidates_count=1 through the same
                                                       #   sequence (Req 1.5); register rejoin callback;
                                                       #   sched_publish_dispatch/sched_publish_pend_reason
                                                       #   gain a retry-until-success-or-shutdown wrapper
                                                       #   around their xcecli_* call (Req 7)
src/scheduler/src/main.c                              # fix hardcoded `if (candidates_count > 3)`
                                                       #   (main.c:314) to reference N_MAX_MASTERS
                                                       #   symbolically, same as worker's fix above;
                                                       #   no ordering change needed (already correct)

src/frontend/apiserver/backend/clib.go                # rewrite RegisterToBackend: builds exactly ONE
                                                       #   xce_cluster_config_t regardless of source
                                                       #   (config.Peers parsed via xce_parse_peers, or
                                                       #   a synthesized single candidate from
                                                       #   config.Address/config.Port when Peers is
                                                       #   empty -- Req 1.5) then the same
                                                       #   validate/xce_cluster_config_to_client_config/
                                                       #   connect sequence either way; deletes
                                                       #   tryConnect()'s direct C.xce_client_config_t
                                                       #   construction and the Go-level per-peer loop
                                                       #   entirely; cgo preamble gains a static C shim
                                                       #   (mirrors the existing
                                                       #   xce_backend_handler/go_push_handler pattern)
                                                       #   that calls the reconnect.go export, passed to
                                                       #   xcecli_set_reconnect_handler
src/frontend/apiserver/backend/peers.go               # DELETED: hand-written parsePeer() retired in
                                                       #   favor of the CGO-routed xce_parse_peers (Req 6.1)
src/frontend/apiserver/backend/health.go              # NEW: backend-degraded circuit-breaker state
src/frontend/apiserver/backend/reconnect.go           # NEW: exported Go rejoin logic (re-sends
                                                       #   xcecli_join for HostRole_apiserver, clears
                                                       #   health.go state); called from clib.go's C
                                                       #   shim AND directly from the heartbeat callback
src/frontend/apiserver/backend/heartbeat.go           # MODIFIED: tick callback sets a needs_rejoin
                                                       #   flag on a failure->success transport
                                                       #   transition (FF_CALL never sets it itself),
                                                       #   then on every subsequent tick where
                                                       #   xcecli_heartbeat succeeds AND needs_rejoin is
                                                       #   still true (not gated on
                                                       #   ApiserverBackendHealth.IsDegraded(), which has
                                                       #   an independent TTL -- Req 2.5) retries
                                                       #   reconnect.go's rejoin function until it
                                                       #   actually succeeds
src/frontend/apiserver/utils/config.go                # no schema change (Peers/Address/Port already exist)

src/worker/testsuite/test_e2e_multimaster_join.c       # NEW: candidate-list join + reconnect E2E;
                                                       #   also covers the unified single-host path
                                                       #   (no --peers) hitting the same code (Req 1.5)
src/scheduler/testsuite/test_e2e_multimaster_join.c    # NEW: same, scheduler
src/frontend/apiserver/backend/health_test.go          # NEW: circuit-breaker unit tests
src/frontend/apiserver/backend/clib_test.go            # + candidate-list parity tests (existing file)

controller/ and libxce's failover mechanism itself are not modified — only consumed.

System Flows

Startup connect (Req 1)

sequenceDiagram
    participant Daemon
    participant Convert as xce_cluster_config_to_client_config
    participant Ctx as xcecli_ctx (client.c)
    participant CandA as Candidate A (standby)
    participant CandB as Candidate B (primary)

    Daemon->>Convert: full candidate list (validated)
    Convert-->>Daemon: xce_client_config_t (nmasters=N)
    Daemon->>Ctx: xcecli_ctx_init + xcecli_start
    Ctx->>CandA: connect
    CandA-->>Ctx: connected, not primary
    Ctx->>Daemon: xcecli_join (role)
    Ctx->>CandA: join request
    CandA-->>Ctx: standby redirect
    Ctx->>Ctx: rotate to next candidate
    Ctx->>CandB: connect + join
    CandB-->>Ctx: joined as primary
    Ctx-->>Daemon: xcecli_join success

Runtime failover (Req 2, Req 3)

sequenceDiagram
    participant Daemon
    participant Ctx as xcecli_ctx lock held
    participant OldPrimary as Old Primary dies
    participant NewPrimary as New Primary

    Daemon->>Ctx: xcecli_* call existing traffic
    Ctx->>OldPrimary: send
    OldPrimary--xCtx: connection lost
    Ctx->>Ctx: XCECLI_FAILOVER_CALL rotates and backs off
    Ctx->>NewPrimary: reconnect
    NewPrimary-->>Ctx: connected
    Ctx->>Daemon: invoke rejoin callback lock still held
    Note over Daemon: callback only sets a needs-rejoin flag, no I/O, returns 0
    Ctx->>NewPrimary: resume original call
    NewPrimary-->>Ctx: reply
    Ctx-->>Daemon: original call result, lock released
    Note over Daemon: after the call returns, lock is free
    Daemon->>Daemon: check and clear needs-rejoin flag
    Daemon->>Ctx: xcecli_join role separate call
    Ctx->>NewPrimary: join
    NewPrimary-->>Ctx: joined
    Ctx-->>Daemon: rejoin complete

Key decision reflected in the diagram: the rejoin callback invoked by the library never performs I/O — it only flags that a rejoin is needed (see "Rejoin callback constraint" above, avoiding a self-deadlock on the still-held per-ctx lock). The actual xcecli_join call happens as a distinct follow-up call from the daemon, on the same thread (worker: connector thread; scheduler: the main driver-loop thread, inside sched_publish_dispatch immediately after xcecli_jobdispatch returns — not the allocator thread, see SchedulerControllerClient below; apiserver: the requesting goroutine), once the triggering call has returned and released the lock — no separate reconnect loop or new thread is introduced either way.

apiserver fail-fast while degraded (Req 4)

sequenceDiagram
    participant Client as API Client
    participant Handler as apiserver handler
    participant Health as health.go
    participant Ctx as xcecli_ctx (gCtx)

    Client->>Handler: HTTP request needing backend
    Handler->>Health: is backend degraded?
    alt degraded
        Health-->>Handler: yes
        Handler-->>Client: 503 service unavailable (fast)
    else healthy
        Health-->>Handler: no
        Handler->>Ctx: backend call
        alt call exhausts retry budget
            Ctx-->>Handler: error
            Handler->>Health: mark degraded (with TTL)
            Handler-->>Client: 503 service unavailable
        else call succeeds
            Ctx-->>Handler: result
            Handler-->>Client: 200 with result
        end
    end

apiserver idle-period reconnect (Req 2.4)

sequenceDiagram
    participant HB as heartbeat.go tick
    participant Ctx as xcecli_ctx (gCtx, FF_CALL)
    participant Rejoin as reconnect.go
    participant Health as health.go
    participant NewPrimary as New Primary

    Note over HB: no client requests in flight
    HB->>Ctx: xcecli_heartbeat (fire-and-forget)
    Ctx->>Ctx: connection known-dead: best-effort rotate + reconnect
    Ctx->>NewPrimary: reconnected transport only
    Ctx-->>HB: rc success (transport reconnected)
    Note over HB: was failing, now succeeds -> set needs_rejoin=true (FF_CALL never sets it for us)
    HB->>HB: needs_rejoin still true?
    HB->>Rejoin: trigger rejoin
    Rejoin->>NewPrimary: xcecli_join(HostRole_apiserver)
    alt join succeeds
        NewPrimary-->>Rejoin: joined
        Rejoin->>HB: clear needs_rejoin
        Rejoin->>Health: MarkHealthy()
    else join's own retry budget exhausted
        NewPrimary--xRejoin: -E_XCE_NOT_READY
        Note over HB: needs_rejoin stays true (independent of Health's TTL-governed state) — next tick retries
    end

This path exists because XCECLI_FF_CALL (fire-and-forget, used by the heartbeat) never invokes the library's own rejoin hook — only the full-retry path (XCECLI_FAILOVER_CALL, used by write calls) does. Without this explicit trigger, an idle apiserver instance would repair its TCP connection on every heartbeat tick but never re-establish cluster membership, so push notifications would stay dead until an unrelated write request happened to arrive.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1-1.3	Try each candidate in order at startup	libxce (unchanged), ConfigConverter	xcecli_start, xcecli_join	Startup connect
1.4	Worker uses full list on FIRST attempt	WorkerControllerConnector	controller_connector.c init order fix	Startup connect
1.5	Single controller is candidates_count=1 of the one mechanism, not a separate fallback (redefined 2026-08-10, was "unchanged single-host mode")	All three	ConfigConverter (nmasters=1 case, no branch) — worker's init_client_config()/single-host branch, scheduler's single-host else branch, and apiserver's tryConnect() all deleted	Startup connect
1.6	Clear startup failure if all candidates fail	All three	existing xcecli_start/xcecli_join return codes	Startup connect
1.7	MAX_HA_CANDIDATES (config.h) is the single source of truth for max candidates; N_MAX_MASTERS derives from it	ConfigConverter (clamp becomes defensive, not normally triggered)	client.h's N_MAX_MASTERS #define	—
2.1-2.3	Automatic reconnect + retry until success or shutdown	libxce (unchanged), all three's rejoin callbacks	xcecli_set_reconnect_handler	Runtime failover
2.4	apiserver resumes push notifications after reconnect (incl. idle-period recovery)	ApiserverBackendBridge	reconnect.go's actual-rejoin function re-sends xcecli_join(HostRole_apiserver), triggered via the shared post-call helper's flag check on write-path reconnect AND explicitly from heartbeat.go on idle-period recovery	Runtime failover, apiserver idle-period reconnect
2.5	Re-join call itself retries on failure, not treated as one-shot	WorkerControllerConnector, SchedulerControllerClient, ApiserverBackendBridge	flag/degraded-state clears only on xcecli_join success, not on mere attempt; apiserver's idle-path checks "still degraded" every tick, not just the failure→success edge	Runtime failover, apiserver idle-period reconnect
3.1-3.3	Retry/backoff doesn't block other liveness work	WorkerControllerConnector (thread isolation, full isolation per AC1), SchedulerControllerClient (bounded, occupancy-dependent buffering via jbuf/decisions cbufs per AC2 — narrower than worker's guarantee by design, no job lost per Req 7, resolved 2026-08-10)	existing thread structure	Runtime failover
4.1-4.2	apiserver fails fast to clients while degraded	ApiserverBackendHealth	health.go	apiserver fail-fast flow
5.1-5.2	Per-candidate connect/reconnect logging	All three	log statements at each rotation	Startup connect, Runtime failover
6.1-6.2	Consistent candidate-list syntax	ApiserverConfigParity	CGO call to xce_parse_peers/xce_cluster_config_validate	Startup connect
7.1-7.6	Scheduler decision publish survives retry-budget exhaustion without loss	SchedulerControllerClient	sched_publish_dispatch/sched_publish_pend_reason retry-until-success-or-shutdown wrapper	Runtime failover

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies	Contracts
ConfigConverter	libxce (C)	Convert validated candidate list to client config	1.1-1.7, 6.1	xce_cluster_config_validate (P0)	Service
WorkerControllerConnector	worker (C)	Single init/connect/join + rejoin for worker	1.1-1.5, 2.1-2.3, 2.5, 3.1, 3.3, 5.1-5.2	ConfigConverter (P0), xcecli_* (P0)	Service
SchedulerControllerClient	scheduler (C)	Single init/connect/join + rejoin for scheduler; decision-publish never silently drops on retry-budget exhaustion	1.1-1.3, 1.5-1.7, 2.1-2.3, 2.5, 3.2-3.3, 5.1-5.2, 7.1-7.6	ConfigConverter (P0), xcecli_* (P0)	Service
ApiserverBackendBridge	apiserver (Go/CGO)	Single init/connect/join for apiserver; config parity	1.1-1.7, 2.4-2.5, 5.1-5.2, 6.1-6.2	ConfigConverter via CGO (P0), xcecli_* via CGO (P0)	Service
ApiserverBackendHealth	apiserver (Go)	Fail-fast circuit breaker for backend-down state	4.1-4.2	ApiserverBackendBridge (P0, consumes rejoin/error signals)	State

Shared Library

ConfigConverter

Field	Detail
Intent	Turn a validated xce_cluster_config_t into a fully-populated xce_client_config_t, clamped to N_MAX_MASTERS with a WARN on truncation
Requirements	1.1, 1.2, 1.3, 1.5, 1.6, 1.7, 6.1

Responsibilities & Constraints

• Pure, socket-free transformation (mirrors xce_test's existing inline logic at test.c:360-372, generalized into one reusable function).
• Does not call xce_cluster_config_validate() itself — callers validate first (existing contract, unchanged) and pass a pre-validated config.
• Does not own from/key (identity/auth) — caller sets those on the output struct after conversion, matching how xce_test, worker, and scheduler already source them independently (CLI/env-specific).
• Handles nmasters=1 as the ordinary case, not a special one (Req 1 AC5): a one-candidate cfg produces a one-master out through the identical copy loop as any other count — no branch distinguishes "single controller" from "multi controller" anywhere in this function.
• The N_MAX_MASTERS clamp is now a defensive safety net, not a normally-triggered path (Req 1 AC7): since client.h's N_MAX_MASTERS now derives from config.h's MAX_HA_CANDIDATES, and xce_cluster_config_validate() already rejects candidates_count > MAX_HA_CANDIDATES before a config ever reaches this function, cfg->candidates_count > N_MAX_MASTERS cannot occur via the validated-input contract this function documents as its precondition. The clamp+WARN stays in place as a guard against a caller that skips validation or a future re-divergence of the two constants — it is intentionally not removed, just no longer load-bearing in the normal path.

Dependencies

• Inbound: WorkerControllerConnector, SchedulerControllerClient, ApiserverBackendBridge (all P0)
• Outbound: none (leaf function)

Contracts: Service [x]

Service Interface

/* xce/cluster.h */
/**
 * xce_cluster_config_to_client_config() — convert a validated candidate list
 * into an xce_client_config_t's master fields (nmasters/masters/master_ports).
 *
 * Copies up to N_MAX_MASTERS candidates from cfg->candidates[] in order.
 * If cfg->candidates_count > N_MAX_MASTERS, copies the first N_MAX_MASTERS and
 * logs a WARN identifying the truncated count (mirrors xce_test's existing
 * clamp behavior).
 *
 * Does NOT set out->from / out->key — caller sets those separately.
 * Does NOT validate cfg — caller must have already called
 * xce_cluster_config_validate() successfully.
 *
 * Returns 0 on success, -EINVAL if cfg or out is NULL or cfg->candidates_count < 1.
 */
int xce_cluster_config_to_client_config(const xce_cluster_config_t* cfg,
                                        xce_client_config_t* out);

• Preconditions: cfg already passed xce_cluster_config_validate().
• Postconditions: out->nmasters in [1, N_MAX_MASTERS]; out->masters[i]/out->master_ports[i] populated for i < out->nmasters, in the same order as cfg->candidates[].
• Invariants: never writes beyond N_MAX_MASTERS entries regardless of cfg->candidates_count.

Worker

WorkerControllerConnector

Field	Detail
Intent	Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect
Requirements	1.1-1.5, 2.1-2.3, 2.5, 3.1, 3.3, 5.1-5.2

Responsibilities & Constraints

• Replaces xcecd_join_cluster's per-candidate loop (controller_connector.c:239-312) with: build ONE xce_cluster_config_t — from cc_ctx->cluster_cfg when --peers was given, or synthesized as a single-candidate xce_cluster_config_t (candidates_count=1, from the existing hardcoded "localhost":9990 default, main.c:333-334) when it was not — run it through xce_cluster_config_validate(), then ConfigConverter, then one xcecli_ctx_init + xcecli_start + xcecli_join. Deletes init_client_config()'s standalone direct-construction of xce_client_config_t (controller_connector.c:129-160) and xcecd_join_cluster()'s separate "single-host fallback (original path)" branch (controller_connector.c:297-311) — both were a wholly parallel mechanism that never touched xce_cluster_config_t/xce_parse_peers/xce_cluster_config_validate at all (confirmed by source audit, 2026-08-10). Per the redefined Req 1 AC5, single-host is not a separately-maintained fallback; it is the candidates_count=1 input to the exact same synthesize-validate-convert-connect sequence multi-candidate configs use.
• Fixes the init-order bug by moving cc_ctx->cluster_cfg population (main.c) to before xcecd_start() is called, so the very first connect already sees the full candidate list (Req 1.4) — this fix composes naturally with the single-path unification above: since there is no longer a separate "first connect uses the single-host branch" code path to accidentally fall into, the only remaining ordering requirement is that whichever xce_cluster_config_t gets built (real --peers list or synthesized single-candidate) is fully assembled before the one synthesize-validate-convert-connect sequence runs, on both the first attempt and any later one.
• Registers a rejoin callback (xcecli_set_reconnect_handler) that only sets a needs_rejoin flag and returns 0 — per "Rejoin callback constraint" above, it must not call xcecli_join (or any xcecli_* function) directly, since the library invokes it while ac->failover.lock is still held.
• checking_connection()'s periodic xcecli_hello keeps its existing role (liveness probe on the connector thread) but no longer needs to manually call xcecd_join_cluster on failure — a transport failure inside xcecli_hello already triggers the library's transparent reconnect + the registered flag-setting callback. Immediately after each xcecli_hello call returns (lock now free), checking_connection() checks needs_rejoin; if set, it calls xcecli_join(ctx, HostRole_worker, &host_attrs) as a separate call — the same join payload used at startup. The flag clears only on that join call's success (Req 2 AC5): if xcecli_join itself returns -E_XCE_NOT_READY (its own retry budget exhausted), needs_rejoin stays set, so the next xcecli_hello tick's return retries the join — checking_connection()'s existing periodic cadence is what makes this retry naturally, no new timer needed.
• Runs entirely on the existing dedicated connector thread (start_controller_connector_thread/connector_to_controller), preserving isolation from the main-loop worker_process_update() and resource_sync_thread liveness paths (Req 3.1) — no new thread introduced.

Dependencies

• Inbound: xcecd_start() (P0)
• Outbound: ConfigConverter (P0), xcecli_* (P0)

Contracts: Service [x]

Implementation Notes

• Integration: preserve existing WARN-level log statements around standby-redirect and candidate-exhaustion, now emitted from the library's classification path or immediately around the single init/join call, so Req 5 diagnostics are not lost in the simplification.
• Validation: implementation must confirm (code read, not test-only) that the connector thread is isolated from the main-loop worker_process_update() and the dedicated resource_sync_thread — the two recurring liveness paths — flagged as a Validation Hook, not re-verified in this design pass. (The three *_update calls previously cited — hardware_monitor_update/system_monitor_update/process_manager_update — are one-shot init calls at xcecd.c:476-480 on the main thread, not recurring liveness duties; corrected per 2026-08-10 gap re-validation.)
• Risks: deleting the hand-rolled loop must not silently drop the MAX_RETRY_COUNT/RETRY_INTERVAL bounded-retry-then-handle_disconnect() behavior in checking_connection() — that outer bound (Req 2.3's "keep retrying" vs. any existing bounded-give-up behavior) must be reconciled explicitly during implementation, not left ambiguous.

Scheduler

SchedulerControllerClient

Field	Detail
Intent	Single connect/join at startup using the full candidate list; transparent reconnect via the library; re-join after reconnect
Requirements	1.1-1.3, 1.5-1.7, 2.1-2.3, 2.5, 3.2-3.3, 5.1-5.2, 7.1-7.6

Responsibilities & Constraints

• Replaces sched_init's inline per-candidate loop (scheduler.c:564-624) with ConfigConverter + one xcecli_ctx_init/xcecli_start/xcecli_join call. Deletes sched_init's "single-host path (original)" else branch (scheduler.c:613, confirmed by source audit 2026-08-10 to be a wholly separate direct-construction mechanism when --peers is not given) — when no --peers is given, sched_init now synthesizes a single-candidate xce_cluster_config_t (candidates_count=1, from init_config()'s existing "localhost":9990 default, main.c:143-152) and runs it through the identical xce_cluster_config_validate() → ConfigConverter → connect sequence as any real --peers list, per the redefined Req 1 AC5: single controller is candidates_count=1 of the one mechanism, not a parallel fallback.
• --sim mode early-returns before sched_init() (main.c:296), so the peers-parsing block only reaches sched_init() for normal-mode operation — the design's assumption that candidate-list wiring always runs before sched_init() in the normal-mode path is confirmed correct; only the line numbers shifted.
• Registers a rejoin callback that only sets a needs_rejoin flag and returns 0 (same constraint as worker — the library invokes it while ac->failover.lock is held, so it must not call xcecli_join itself). Corrected call site (supersedes an earlier, incorrect claim that this runs on sched_allocator): xcecli_jobdispatch is called from sched_publish_dispatch (scheduler.c:1257), which is reached only via the decision-drain loop in xce_sched_driver_run_normal_mode_round (driver.c:765-786) — a loop that runs on the single main-loop thread (main.c's while(1), main.c:348-349), not on sched_allocator. sched_allocator only pushes decisions onto the decisions cbuf (scheduler.c:336,387); it never calls xcecli_jobdispatch directly. The check-needs_rejoin-then-xcecli_join call must therefore live in sched_publish_dispatch, immediately after its xcecli_jobdispatch call returns (lock released) — same thread, same call frame — not in sched_allocator. The flag clears only once that xcecli_join call succeeds (Req 2 AC5): on its own -E_XCE_NOT_READY, needs_rejoin stays set, so the next decision's publish-return retries the join — since Req 7's retry-until-success wrapper means sched_publish_dispatch returns (successfully) far more reliably now than before, this composes cleanly rather than needing its own separate retry loop.
• Corrected thread/backpressure model (Critical Issue from 2026-08-10 design validation, resolved by narrowing below — supersedes the prior claim that sched_job_picker runs "unaffected"): the scheduler's real pipeline is three stages joined by two cbufs: sched_job_picker → jbuf (capacity 1, core.c:161 — the scheduler-core determinism invariant, never resizable without re-deriving scheduler-core's Req 5 AC7) → sched_allocator → decisions (capacity ndecisions=5, core.c:84) → the main-loop thread's decision-drain loop → xcecli_jobdispatch. When that call blocks inside XCECLI_FAILOVER_CALL's retry/backoff (lock held), the main thread stops draining decisions. sched_allocator can still produce more decisions before its own cbuf_put(&core.decisions, ...) blocks, and sched_job_picker can still produce jobs into jbuf for as long as allocator keeps draining it. So picker and allocator are buffered, not isolated: they continue briefly past the point the outage begins, then both stall until connectivity (and dispatch) resumes. The exact size of that grace window is occupancy-dependent, not a fixed constant: if decisions had Q items queued (1≤Q≤5) at the moment the main thread's pop grabs the decision that turns out to be un-publishable, allocator can push (6-Q) more decisions before it stalls, and picker's total new picks during the window come out to (7-Q) — ranging from 2 (Q=5, decisions was already near-full, common under load) up to 6 (Q=1, decisions was nearly empty). Earlier text in this section claimed "at least 6" as a guaranteed floor — that was backwards: 6 is the best-case ceiling (ndecisions+1), the guaranteed floor is as low as 1-2 depending on jbuf's own occupancy at the same instant. This is a materially different, weaker guarantee than "unaffected," and a materially different, more accurate one than the original "at least 6" claim. This "stall until resumes" description is only actually true once the Req 7 fix below is in place — see that bullet for why, unfixed, the main thread can instead silently unstick itself by dropping a decision rather than genuinely staying stalled.
• Decision-publish retry-until-success-or-shutdown wrapper (Req 7, gap found in 2026-08-10 design validation): xcecli_jobdispatch/xcecli_set_pend_reason are XCECLI_FAILOVER_CALL-wrapped, which retries a single call up to a bounded budget (max_retries=20, backoff capped at max_retry_ms=5000ms — roughly 1-2 minutes total, client_failover.c:281-282,312-317) before returning -E_XCE_NOT_READY. Today's driver decision-drain loop (driver.c:783-789) does not inspect this return code — it logs and advances to the next queued decision regardless of outcome. Since sched_commit_decision already mutates scheduler-local job state (job->allocs, accounts) before the publish attempt (scheduler.c sched_allocator), a budget-exhausted publish today silently and permanently diverges scheduler-local state from what the controller was told — not a delay, a loss. Fix: sched_publish_dispatch/sched_publish_pend_reason (scheduler.c:1257/1251) wrap their xcecli_* call in while (!sched->core.stop) { rc = XCECLI_FAILOVER_CALL(...); if (rc != -E_XCE_NOT_READY) break; WARN(extended-retry, ...); } — checking core.stop before each attempt, not only after (a plain do/while checking the stop condition only at the bottom would let every already-queued decision behind the one currently stuck each burn one full ~100s budget cycle discovering shutdown was requested, compounding to minutes-per-queued-decision instead of a single bounded delay — caught in the 2026-08-10 design validation's third pass). Re-invoking the whole macro on each loop iteration resets its own internal retry state and starts a fresh budget, rather than adding a second retry mechanism. This is a straightforward extension of Req 2 AC3's existing "keep retrying, never permanently give up" language, applied per-decision instead of only at the connection level. Shutdown interaction: stop_scheduler()'s existing cbuf_tryput(&core.decisions, NULL) wake-up (main.c:163) only unblocks a thread parked in cbuf_get(&core.decisions) — it does nothing for a thread already inside this new retry loop, so the loop's own core.stop check is the only shutdown path. With the check moved to the top, only the ONE decision actively in-flight when shutdown is requested can cost up to one full budget cycle (~1-2 minutes); every decision still queued behind it in decisions sees core.stop already true and returns immediately without attempting a call, so total shutdown latency stays bounded to a single cycle regardless of queue depth — an accepted bounded tradeoff (same posture as scheduler-core's accepted 5s OpCode_schedAdmin blocking-wait case) since libxce's retry macro itself is out of this spec's boundary to modify for a faster mid-attempt shutdown check. Ordering: decisions remain published strictly in decisions cbuf FIFO order (Req 7 AC3) because the drain loop still only advances to cbuf_get() the next decision after the current one's publish call returns — a stuck decision blocks the whole loop, it does not get skipped and revisited later. Rejoin-check interaction: the existing check-needs_rejoin step (Responsibilities bullet above) sits after this whole while loop, not after each inner attempt — it observes the loop's final, successful return, exactly the same as it would for a single non-retried call.

Dependencies

• Inbound: sched_init() (P0)
• Outbound: ConfigConverter (P0), xcecli_* (P0)

Contracts: Service [x]

Implementation Notes

• Integration: preserve the existing WARN/INFO log lines around candidate parsing (main.c) and add per-attempt/per-outcome logging around the single init/join call (today's loop logged each attempt; the library's own classification path should be checked for equivalent visibility, else add it locally).
• Req 3 AC2 scope note (Critical Issue from 2026-08-10 design validation, resolved 2026-08-10): the original merged AC1 wording ("shall continue performing ... allocation/decision processing ... without being blocked") was not met unconditionally by the scheduler — only for a bounded, occupancy-dependent grace window (floor as low as 1-2 jobs, ceiling ndecisions+1=6, see the corrected backpressure-model bullet above), after which sched_allocator and then sched_job_picker do block. jbuf's capacity is a fixed invariant owned by the scheduler-core spec (out of this spec's boundary to change) and decisions' capacity is an existing config constant; redesigning either is out of scope here. Resolution: Requirement 3's single merged AC1 was split into a worker-only AC1 (unconditional, unchanged guarantee) and a scheduler-only AC2 (bounded — "continues briefly ... then blocks until connectivity is restored, at which point processing resumes automatically ... no job picked or decided during that bounded window is ever lost"), with the former heartbeat AC renumbered to AC3. requirements.md deliberately does not hardcode a specific job count — see the corrected backpressure-model bullet above for why that count is occupancy-dependent (floor 1-2, ceiling ndecisions+1=6), not a fixed constant an initial draft of this note incorrectly stated as a guaranteed "at least 6." requirements.md reflects this split and the corrected framing; the design and its bounded-buffering behavior now match the requirement it's held to, with Requirement 7 supplying the actual load-bearing guarantee (no loss, regardless of how many jobs the buffer absorbs).
• Risks: today's total absence of reconnect logic means this is additive behavior change (scheduler previously required a manual restart after primary loss) — call out explicitly in the task/rollout notes so operators aren't surprised by scheduler now retrying indefinitely instead of exiting.

apiserver

ApiserverBackendBridge

Field	Detail
Intent	Single connect/join at startup using the full candidate list via the C library (through CGO), replacing the Go-level per-peer loop; config-syntax parity with scheduler/worker
Requirements	1.1-1.7, 2.4-2.5, 5.1-5.2, 6.1-6.2

Responsibilities & Constraints

• RegisterToBackend() (clib.go) is rewritten to build exactly ONE C xce_cluster_config_t regardless of source, then run it through the single synthesize-validate-convert-connect sequence worker/scheduler use: if config.Peers is non-empty, format it into the same [name=]host:port[,...] syntax and parse via xce_parse_peers (CGO, not the existing hand-written Go parsePeer); if empty, synthesize a one-candidate xce_cluster_config_t (candidates_count=1) directly from config.Address/config.Port — either way the result is validated with xce_cluster_config_validate, converted with ConfigConverter, then xce_backend_init is called once with the resulting config (nmasters in [1, N_MAX_MASTERS], not necessarily >1). Deletes tryConnect()'s direct C.xce_client_config_t construction (clib.go:79-101) and the for _, peer := range config.Peers { tryConnect(...) } Go-level loop (clib.go:116-135) entirely, along with parsePeer() (backend/peers.go:11, retired per Req 6.1) — both tryConnect()'s Peers-loop call site and its no-Peers Address/Port call site (clib.go:138) were separate mechanisms that never touched xce_cluster_config_t at all (confirmed by source audit, 2026-08-10); per the redefined Req 1 AC5, a bare Address/Port config is candidates_count=1 of the same mechanism, not a separate fallback.
• reconnect.go holds two plain Go functions: (1) a //export-annotated flag-setter — registered as the library's rejoin callback, does no I/O, only sets a needs_rejoin atomic flag and returns 0 (per "Rejoin callback constraint" above: the library invokes it while ac->failover.lock is held, so it must not call xcecli_join/any CGO backend function itself); (2) a plain Go function that performs the actual rejoin — calls xcecli_join(gCtx.ctx, HostRole_apiserver, NULL) and clears ApiserverBackendHealth's degraded state on success.
• The flag-setter is registered with the C library from clib.go's own cgo preamble, via a small static C shim (xce_backend_rejoin_handler, mirroring the existing xce_backend_handler/go_push_handler pattern already used for push dispatch) passed to xcecli_set_reconnect_handler — cgo symbol resolution is per-file (see wire_dispatch_support.go's documented rationale for the identical go_push_handler wrapper), so the shim and the registration call must live in the same file as the extern declaration, not standalone in reconnect.go.
• Rather than duplicating the same pre-call/post-call logic across all four wrapper functions, the pre-call IsDegraded() fail-fast check (ApiserverBackendHealth, below) and the post-call needs_rejoin check are both implemented once, in a single shared internal helper each of the four clib.go wrappers calls around its C.xcecli_* invocation — not copy-pasted per wrapper. The flag clears only once the follow-up xcecli_join call succeeds (Req 2 AC5): if that call itself returns -E_XCE_NOT_READY, needs_rejoin stays set so the next backend-call wrapper's return retries the join, and MarkHealthy() is correspondingly only called on that eventual success — not on the mere attempt.
• Idle-period reconnect gap (Req 2.4) fix: XCECLI_FF_CALL — the macro backing xcecli_heartbeat, heartbeat.go's existing periodic prober (cmd/main.go:76) — performs a best-effort transport reconnect on a known-dead connection but never invokes the library's rejoin hook (verified against client_internal.h/client_failover.c: xcecli_invoke_rejoin is called only from xcecli_retry_next, the full-retry path used by write calls, never from the fire-and-forget path). Left as-is, a controller failover occurring while apiserver has no in-flight write request would silently repair the TCP connection without ever re-joining, permanently starving push notifications until an unrelated write request happens to arrive. Fix: on a failure→success transport transition, heartbeat.go's tick callback sets the same needs_rejoin atomic flag the write-path's C-side callback sets — since XCECLI_FF_CALL never invokes that callback for this path, heartbeat.go sets the Go-side flag itself. On every subsequent tick where xcecli_heartbeat succeeds and needs_rejoin is still true (checked every tick, not only on the transition edge — Req 2 AC5 gap found in the 2026-08-10 design validation's second pass), it calls reconnect.go's rejoin function; the flag (and ApiserverBackendHealth's degraded state) clears only when that rejoin call itself succeeds. Deliberately not gated on ApiserverBackendHealth.IsDegraded() (a design-validation third-pass correction over an earlier draft of this fix): that state has its own independent TTL-based auto-clear (see ApiserverBackendHealth below) for request-path circuit-breaker purposes — if the TTL expired before the rejoin actually succeeded, gating the retry on "still degraded" would stop retrying the join even though it never succeeded, silently reintroducing the same starvation this fix exists to close. needs_rejoin and ApiserverBackendHealth's degraded-until timestamp are two independent pieces of state serving two different purposes (one tracks "have I re-joined yet," the other "should request handlers fail fast right now") and must not be conflated. A transition-edge-only trigger would fire the rejoin attempt exactly once per outage and, if that one attempt's own xcecli_join hit its retry budget and failed, never retry — every subsequent tick would see success→success (transport already up) and never re-fire; checking the persistent flag instead of the transient edge makes each tick a natural retry opportunity for as long as apiserver remains un-joined. This reuses the same rejoin logic Task 4.3 already builds — no duplicate join-payload code.

Dependencies

• Inbound: cmd/main.go (P0)
• Outbound: ConfigConverter via CGO (P0), xcecli_* via CGO (P0), ApiserverBackendHealth (P1, signals it on outcome)

Contracts: Service [x]

Service Interface

// backend/clib.go
// RegisterToBackend initialises the client using the full configured
// candidate list — config.Peers if set, otherwise a single-candidate list
// synthesized from config.Address/config.Port (not a separate code path,
// see Req 1 AC5) — and registers the rejoin callback. Single call — no
// per-peer Go loop.
func RegisterToBackend(config *BackendConfig) error

• Preconditions: StartNotificationListener already called (unchanged from today).
• Postconditions: on success, gCtx.ctx is joined with nmasters reflecting every configured candidate (clamped per ConfigConverter); the rejoin callback is registered.
• Invariants: candidate-list parsing/validation errors surface the same way xce_parse_peers/xce_cluster_config_validate report them in C (Req 6.2) — not a separately-worded Go error.

Implementation Notes

• Integration: parsePeer (Go) and its tests are removed/retired in favor of the CGO-routed parser (Req 6.1).
• Integration: the rejoin C shim must be defined in clib.go's cgo preamble (not reconnect.go) — see Responsibilities above; this was the design's one build-risk item, now resolved by construction rather than left as an implementation-time surprise.
• Risks: CGO error surfaces (C -EINVAL etc.) need a clear Go-side error message mapping so operators get an actionable message (Req 1.6, 6.2), not a bare error code.

ApiserverBackendHealth

Field	Detail
Intent	Let request handlers fail fast (503) instead of blocking on the library's full retry budget when the backend is known-degraded
Requirements	4.1, 4.2

Responsibilities & Constraints

• New health.go: an atomic "degraded-until" timestamp. IsDegraded() bool — true if now < degraded-until. MarkDegraded(ttl time.Duration) — called by backend-call wrapper functions in clib.go when a call's returned error indicates retry-budget exhaustion (e.g. E_XCE_NOT_READY). MarkHealthy() — called by the rejoin callback (reconnect.go) on a successful rejoin.
• Backend-call wrapper functions used by request-handling code paths (job submit, job control, queue control, host control, etc. — the existing gCtx.ctx-based calls in clib.go) check IsDegraded() first, via the shared internal helper described under ApiserverBackendBridge above (not duplicated per wrapper); if true, return a clear "backend unavailable" error immediately instead of invoking the blocking C call (Req 4.1). Calls that don't require the backend are unaffected (Req 4.2 — nothing to change there, they already don't touch gCtx).

Dependencies

• Inbound: ApiserverBackendBridge (sets state on outcomes)
• Outbound: none

Contracts: State [x]

State Management

• State model: single atomic degraded-until timestamp (int64 unix-nano via sync/atomic, same pattern as existing heartbeatIntervalSecs in heartbeat.go).
• Persistence & consistency: process-local, in-memory only; reset on apiserver restart (matches the rest of apiserver's in-memory state model per tech.md).
• Concurrency strategy: lock-free atomic read/write; safe for concurrent goroutines per request.

Implementation Notes

• Integration: TTL-based clearing (Risks & Mitigations in research.md) in addition to the rejoin-callback clear, so a missed/edge-case rejoin doesn't permanently wedge the flag.
• Validation: unit tests cover degraded→healthy transitions via both the TTL expiry and explicit MarkHealthy().
• Deliberately independent of needs_rejoin (Req 2 AC5, correction from the 2026-08-10 design validation's third pass): IsDegraded()'s TTL-based auto-clear exists so request handlers eventually probe again even without an explicit MarkHealthy() — a legitimate circuit-breaker half-open pattern. It must not be used as the retry-trigger condition for the heartbeat-idle-path's rejoin attempts (ApiserverBackendBridge above) — the TTL could expire before an actual rejoin succeeds, and gating the rejoin retry on "still degraded" would then silently stop retrying the join while apiserver is still not actually joined. The rejoin-tracking needs_rejoin flag has no TTL and clears only on an actual xcecli_join success, independent of whatever ApiserverBackendHealth's degraded-until timestamp is doing.

Error Handling

Error Strategy

No new error taxonomy — reuse the existing controller status codes (E_XCE_REDIRECT/E_XCE_STANDBY/E_XCE_NOT_READY, errnox.h) that client_failover.c already classifies, and the existing xce_cluster_config_validate() -EINVAL cases for malformed candidate-list config.

Error Categories and Responses

• Malformed candidate-list config (Req 1.6, 6.2): xce_parse_peers/xce_cluster_config_validate failure at startup → each component reports a clear message naming the offending entry and refuses to start (unchanged behavior for worker/scheduler; apiserver gains the same via the CGO route).
• All candidates unreachable at startup (Req 1.6): xcecli_start/xcecli_join returns the last transport error after exhausting all candidates → component reports which candidates were attempted (Req 5.2) and fails startup (worker/scheduler: process exits as today; apiserver: RegisterToBackend returns an error, cmd/main.go's existing handling applies unchanged).
• Runtime disconnect (Req 2): transparently retried/rotated by the library; components only observe it through their own periodic calls' return codes and the rejoin callback — no new error category, just new logging around the existing outcomes.
• Scheduler decision-publish retry-budget exhaustion (Req 7): -E_XCE_NOT_READY from xcecli_jobdispatch/xcecli_set_pend_reason is not treated as terminal — the caller re-invokes the same publish call rather than surfacing an error or dropping the decision (see SchedulerControllerClient's retry-until-success-or-shutdown wrapper). This is scheduler-specific: worker's periodic xcecli_hello carries no payload to lose on the same error, and apiserver's request/response calls correctly surface -E_XCE_NOT_READY to ApiserverBackendHealth since there the caller (the original HTTP client) is expected to retry, unlike scheduler's decisions which have no external retry source.
• Candidate count exceeds N_MAX_MASTERS: ConfigConverter clamps and WARNs (not an error) — matches xce_test's existing behavior exactly.
• apiserver backend degraded (Req 4.1): mapped to a clear 503-style API error, distinct from other 5xx causes, so operators/clients can distinguish "backend down, retrying" from an unrelated internal error.

Monitoring

Per-candidate connect/reconnect attempts and outcomes are logged (Req 5) using each component's existing logger (XCECD_LOG for worker, the scheduler's existing INFO/WARN macros, zap for apiserver) — no new logging infrastructure, just new call sites around the simplified connect/reconnect path.

Testing Strategy

• Unit Tests:
  • ConfigConverter: clamps at N_MAX_MASTERS with WARN when given more candidates; preserves candidate order; rejects NULL/empty input (test_cluster.c); a candidates_count=1 input produces nmasters=1 through the identical code path as any other count (Req 1.5) — no branch, no special case.
  • N_MAX_MASTERS == MAX_HA_CANDIDATES (Req 1.7): a _Static_assert (or equivalent compile-time check) in client.h or a test_cluster.c case asserting the two constants stay equal — a regression guard against reintroducing the divergence this design pass reconciled. Also re-run test_libxce_retry.c's fo_make_client_masters() fixture with its corrected slot count and confirm it doesn't silently exercise an uninitialized masters[3].
  • apiserver health.go: degraded→healthy via MarkHealthy(), via TTL expiry; concurrent goroutine safety (health_test.go).
  • apiserver CGO-routed parser: same malformed-input cases scheduler/worker already reject, now asserted through the Go call site (clib_test.go).
  • Rejoin flag-setter callbacks (worker, scheduler, apiserver): with a real ac->failover.lock held (simulating the library's call context per "Rejoin callback constraint"), invoking the registered callback returns promptly without blocking and performs no I/O — a regression guard against reintroducing the self-deadlock this design pass found.
  • Rejoin retry-on-failure (Req 2 AC5, worker/scheduler/apiserver): using a test double for xcecli_join that fails N times before succeeding, assert the pending-rejoin indicator (flag or degraded-state) stays set/degraded after each failed attempt and is cleared only on the eventual success; assert the next naturally-occurring opportunity (next xcecli_hello/decision-publish/backend-call return, or next heartbeat tick for apiserver's idle path) is what re-attempts it, not a dedicated new retry loop — a regression guard against the transition-edge/clear-before-success versions of this bug found in the 2026-08-10 design validation's second pass.
• Integration Tests:
  • Scheduler decision-publish redelivery (Req 7): using a test double for xcecli_jobdispatch/xcecli_set_pend_reason that returns -E_XCE_NOT_READY for N calls before succeeding, assert the same decision is retried (not dropped, not skipped) and eventually published; assert decisions queued behind it publish afterward in original order (Req 7 AC1-3); assert setting sched->core.stop mid-retry breaks the loop within one budget cycle (Req 7 AC2); assert a WARN-level log line distinguishes extended-retry success from first-attempt success (Req 7 AC4); with multiple decisions queued (decisions at or near its 5-item capacity) when core.stop is set mid-retry on the first, assert total shutdown latency stays bounded to roughly one budget cycle, not one per queued decision — a regression guard against the compounding-shutdown-latency bug found in the 2026-08-10 design validation's third pass.
  • Unified single-host path (Req 1.5): starting each of worker, scheduler, and apiserver with no --peers/empty backend.peers reaches the identical xce_cluster_config_validate → ConfigConverter → connect call sequence as a single-entry --peers config — assert via the same log-line signature both paths now share (there is no longer a distinct "single-host mode" log message), not just "it connects."
  • Worker/scheduler startup against a 3-candidate config where candidate[0] is a live standby — assert the component joins as expected without operator intervention (Req 1.1-1.3), extending each component's testsuite/.
  • Worker/scheduler reconnect: kill the currently-joined primary mid-session, assert the component reconnects and re-joins the new primary without restart (Req 2.1-2.3), reusing the project's existing /xce-deploy-ha + /xce-test-ha pattern already used for controller HA soak testing — this is also the test that would hang (not fail cleanly) if the self-deadlock were reintroduced, so it must run with a wall-clock timeout that fails loud rather than hanging CI.
• E2E Tests:
  • Full 3-node HA cluster with worker + scheduler + apiserver all configured with the full candidate list; kill the primary; assert all three reconnect, worker resumes job execution reporting, scheduler resumes dispatch, apiserver resumes serving requests and receiving push notifications (Req 2.4) — new test_e2e_multimaster_join.c per C component plus an apiserver-side check in the existing HA soak harness.
  • apiserver idle-period reconnect: kill the primary while apiserver has no in-flight write request (only the heartbeat prober running); assert apiserver still rejoins and resumes push notifications, proving the heartbeat-triggered rejoin path (not just the write-call-triggered one) works (Req 2.4).
  • apiserver fail-fast: with the backend killed and not yet reconnected, assert an API request returns quickly (bounded, sub-second) with a clear 503 rather than hanging for the full retry budget (Req 4.1).

Security Considerations

No new trust boundary: candidate hosts are the same set of controllers already trusted today; TLS/identity handling is unchanged (see Non-Goals — CA-cert wiring is out of scope and pre-existing). No new authentication surface is introduced.