myxce/.kiro/specs/libxce-failover/design.md

Design Document — libxce-failover

Overview

Purpose: This feature delivers transparent controller-primary failover to every client application built on libxce (worker, scheduler, apiserver via cgo, and the xce_test harness), so that a controller failover is hidden when short and reported clearly when not.

Users: libxce callers issue normal requests (xce_joblist, xce_jobsub, …). When the connected primary dies, libxce locates the newly elected primary and completes the request there, or — if no primary exists yet — returns a distinct cluster-not-ready error instead of an opaque EINVAL. No caller code changes.

Impact: Today the client connects to one master (client.c:122), routes every request through the fire-and-forget xce_request(), and has no reply-status-driven retry on the production path (xce_request_with_retry() only retries in test-hook mode). This design places failover at the client.c (xcecli_*) layer — the universal client surface that every consumer goes through (worker, scheduler, apiserver-via-cgo, xce_test) and the only layer whose context (api_ctx_t) co-locates the master list (conf), the endpoint (ep), and the reply context (ctx). Each xcecli_* op already calls its internal xce_* op, which returns the controller status; the failover loop wraps that call, classifies the status, and reconnects/rotates over api_ctx.conf. It adds master-list rotation, redirect-address following, and a small controller change so redirect replies carry the primary address. xceapi.c ops are unchanged.

Goals

• Route all libxce request/reply round trips through one transparent failover path (1.1, 1.3).
• Discover the new primary via redirect address, falling back to master-list rotation (2.x, 8.x).
• Hide short failovers via bounded backoff; surface a distinct cluster-not-ready on exhaustion (3.x).
• Fail over reads and writes uniformly, relying on controller idempotency for writes (5.x).
• Reuse the existing retry config/defaults; stagger retries with jitter (6.x).

Non-Goals

• Controller-side election, WAL, commit barrier, write-idempotency logic (consumed, not changed).
• Per-daemon adoption (master-list wiring, liveness-safe blocking) — component-failover-adoption.
• New transport opcodes or new reply fields (redirect fields already exist).

Boundary Commitments

This Spec Owns

• The client-side failover loop: classify status → reconnect/rotate/backoff → resume, including the cluster-not-ready terminal mapping, for reply-bearing xcecli_* ops.
• Client-side master-list cursor and current-primary stickiness within a context.
• Capturing the redirect address from a reply as a best-effort hint (no correctness dependency).
• A reconnect/rejoin hook: a consumer-registered callback the failover machinery invokes after it reconnects to a new primary and before it resumes the interrupted op. libxce owns the hook mechanism, its ordering, and invoking it; it does not own the callback's content.
• A per-ctx concurrency lock so xcecli_* failover is safe when one ctx is shared across threads.
• The single controller change that populates redirect_host/port on E_XCE_REDIRECT replies.
• Realignment of the libxce retry unit tests to the real controller status taxonomy.

Out of Boundary

• How the controller decides primary/standby/redirect/no-primary (election, ha_should_redirect).
• The content of the rejoin callback: the join/state-exchange a daemon performs on reconnect (owned by component-failover-adoption) and the worker's running-jobs re-sync (owned by worker-rejoin-resync). This spec provides and invokes the hook; the daemons fill it in.
• Redelivery of fire-and-forget notifications lost across a failover (also worker-rejoin-resync).
• Worker/scheduler/apiserver adoption and their threading/liveness handling of bounded blocking.
• Any reply payload beyond the already-defined redirect_host/redirect_port.

Allowed Dependencies

• Existing controller status codes (errnox.h:22-25) and redirect reply fields (typemap.h:225).
• endpoint_connect() / endpoint I/O (netio.*), request encode/enqueue (request.c), and the context wait channel (cbuf, xcectx.c).
• xce_retry_config_t + XCE_RETRY_DEFAULTS (request.h) and the cluster/peer config (cluster.*).

Revalidation Triggers

• Change to the reply status/redirect_* contract → re-check client classification + controller.
• Change to the async reply-dispatch model (handle_reply → cbuf signalling) → re-check round trip.
• Change to api_ctx_t/endpoint ownership → re-check reconnect-reuses-same-endpoint assumption.
• New non-primary status code added to the controller → extend client classification.
• Change to the reconnect/rejoin hook signature or its invocation ordering → re-check daemon rejoin registration (component-failover-adoption) and worker resync (worker-rejoin-resync).

Architecture

Existing Architecture Analysis

• Layering — xcecli_* is the universal surface. Every consumer calls client.c's xcecli_* ops, never xceapi.c's xce_* directly: worker (controller_connector.c:184,272), scheduler (scheduler.c:457,463), apiserver via cgo (clib.go:34,36), xce_test. Each xcecli_* is a thin wrapper that calls its xce_* counterpart, which returns the controller status as its rc.
• Only api_ctx_t reaches the master list. api_ctx_t = { conf (masters), ep, ctx } (client.c:23). The inner xce_ctx_t (xceapi) holds ep/wbuf/state but no conf (xcectx.h) — so failover belongs at the client layer, where the master list is already in scope.
• xce_* fail fast; xcecli_* retries. The async send/wait stays inside xce_* unchanged: a single attempt that returns the controller status (or transport error). The xcecli_* wrapper classifies that rc and, on a retriable status, reconnects/rotates api_ctx.ep over api_ctx.conf and re-invokes xce_*. No async internals, thunks, or master-list plumbing are introduced.
• Reconnect reuses the endpoint. endpoint_connect(ep,host,port) re-points the same endpoint_t, preserving the msg_handler wiring that routes replies to ctx->wbuf.
• Stateful consumers need a rejoin on reconnect. xce_test is a pure client — reconnect + resend suffices. But the apiserver, worker, and scheduler hold a session with the primary established via xcecli_join (role + attrs); a freshly elected primary has no live session for a reconnecting daemon until it re-joins, and a worker must additionally re-sync its running-jobs state. So failover for these consumers is reconnect → rejoin/state-exchange → resume, not just reconnect → resend. libxce models this with a consumer-registered reconnect hook; the hook content lives in component-failover-adoption (join) and worker-rejoin-resync (job state).
• Two op shapes. Reply-bearing ops (xce_joblist/xce_jobsubmit/xce_hello/…) do xce_*_req + cbuf_get → return a controller status the loop can classify. Fire-and-forget ops (xce_notify*, heartbeat) do xce_*_req only (no cbuf_get, xceapi.c:282) → return the enqueue result, never a controller status. Failover classification applies only to the former.
• Patterns preserved: xce_* ops, xce_*_req enqueue wrappers, and xce_request/xce_reply are all unchanged; failover is additive in xcecli_*. Trade-off: ~33 xcecli_* wrappers each gain a small retry loop (more lines), but the policy lives in one shared decision helper (fewer moving parts) — an explicit simplicity-over-LOC choice.

Architecture Pattern & Boundary Map

graph LR
    Caller[consumer worker scheduler apiserver xce_test] --> Cli[xcecli op in client.c]
    Cli --> Loop[do while retry loop]
    Loop --> Decide[xcecli_retry_next decision helper]
    Loop --> Xce[xce op in xceapi one shot]
    Xce --> Ctrl[controller]
    Ctrl --> Reply[handle_reply]
    Reply --> Capture[capture redirect hint into ctx]
    Decide --> Rotate[reconnect or rotate over api_ctx conf and ep]
    Decide --> Backoff[jittered backoff]

• Selected pattern: a thin retry loop in each xcecli_* wrapper, driven by one shared decision helper (xcecli_retry_next). Rationale: client.c is the universal surface AND the only layer whose api_ctx_t co-locates master list + endpoint + ctx; xce_* already returns the status.
• Dependency direction: types → cluster/config → endpoint(netio) → xceapi(xce_*) → client(xcecli_*). Each layer imports only leftward. Failover logic is added at the client layer (rightmost), depending on xce_* and cluster/config; xceapi does not depend on it.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Client wrappers	C (gnu17), libxce client.c	xcecli_* retry loop + decision helper + rotate	Reuses xce_*, endpoint_connect, cluster
Client API	C (gnu17), libxce xceapi.c	Unchanged — one-shot xce_* returns status	No edits
Controller	C (gnu17)	Populate redirect address on E_XCE_REDIRECT	~2 lines in join.c; fields already in schema
Tests	GoogleTest (testsuite)	Realign retry tests to real status taxonomy	Test the xcecli_* retry path

File Structure Plan

New Files

src/lib/client_failover.c       # xcecli_retry_next decision helper, StatusClassifier, rotate/reconnect over api_ctx
src/lib/client_internal.h        # private: api_ctx_t + xce_failover_state_t shared by client.c and client_failover.c

api_ctx_t is currently file-local to client.c; move its definition to client_internal.h so the failover helper can operate on it without exposing it in the public API.

Modified Files

• src/lib/client.c — wrap each reply-bearing xcecli_* op body in a do { rc = xce_<op>(...); } while (xcecli_retry_next(ac,&rc,is_write,&st)); loop (per OpCategory); fire-and-forget ops get the transport-reconnect-only variant; add xce_failover_state_t to api_ctx_t (lock + cursor + retry config + rejoin hook); add xcecli_set_reconnect_handler(); init + take failover.lock per op; init in xcecli_start(); keep initial connect to masters[0].
• src/include/client.h — declare xcecli_set_reconnect_handler() + xcecli_reconnect_cb.
• src/lib/xcectx.c — handle_reply() copies the redirect-hint address (reply.redirect_host/port, when present) into xce_ctx_t before the reply is freed; the existing error path (XCE_CTX_ERR + wbuf error inject, xcectx.c:137-154) is the wake hook reused on reconnect.
• src/include/xcectx.h — add the fixed-size redirect-hint host/port fields to xce_ctx_t.
• src/include/request.h / src/lib/request.c — keep xce_retry_config_t + XCE_RETRY_DEFAULTS; remove the orphaned no-op xce_request_with_retry() (superseded by the xcecli_* loop).
• src/lib/xceapi.c — unchanged (one-shot xce_* ops already return the controller status).
• src/controller/join.c — populate reply.redirect_host = c->ha.primary_host / reply.redirect_port = c->ha.primary_port on the E_XCE_REDIRECT path (audit for other sites).
• src/lib/testsuite/test_libxce_retry.c — retarget from the retired primitive to the xcecli_* retry path + StatusClassifier; cover E_XCE_REDIRECT/NOMASTER/NOT_READY/STANDBY, master rotation, and budget→not-ready.

System Flows

Failover loop (per xcecli_* op)

sequenceDiagram
    participant C as Caller
    participant F as xcecli op do while loop
    participant E as Endpoint
    participant K as Controller
    C->>F: xcecli op
    loop until success, terminal, or budget exhausted
        F->>E: xce op one shot
        E->>K: request
        K-->>E: reply status plus optional redirect addr
        E-->>F: xce op returns status
        alt status success
            F-->>C: ok
        else redirect or standby or transport error
            F->>E: rotate (hint address first if set, else next master) then invoke rejoin hook
        else no-primary or not-ready
            F->>F: backoff jittered then rotate
        else non-retriable
            F-->>C: surface error
        end
    end
    F-->>C: cluster-not-ready on budget exhaustion

After any reconnect to a new endpoint, the failover machinery invokes the consumer-registered reconnect/rejoin hook (if set) before re-sending the interrupted op, so a stateful daemon re-establishes its session (and a worker re-syncs job state) against the new primary. Pure clients (xce_test) register no hook and skip straight to re-send.

Key decisions: redirect/standby/transport rotate without the no-primary backoff delay (4.2); only no-primary/not-ready consume the backoff budget (3.1, 4.1); on success the served master becomes the sticky current primary (2.4).

Status classification

stateDiagram-v2
    [*] --> Classify
    Classify --> Success: status==0 or DUP on re-sent write
    Classify --> RetryNow: REDIRECT or STANDBY or transport error
    Classify --> RetryBackoff: NOMASTER or NOT_READY
    Classify --> Terminal: AUTH PERM INVALID NOENT
    RetryNow --> Rotate
    RetryBackoff --> Rotate
    Rotate --> Classify: hint first else next master
    Success --> [*]
    Terminal --> [*]

Requirements Traceability

Requirement	Summary	Components	Flows
1.1, 1.3	All ops through failover; preserve API	XcecliRetryLoop (all xcecli_*)	Failover loop
1.2	Complete next request against new primary	XcecliRetryLoop, EndpointRotator	Failover loop
2.1	Unreachable → try other masters	EndpointRotator	Classification
2.2, 8.3, 8.4	Redirect addr if present else rotate	RedirectCapture, EndpointRotator	Classification
2.3, 4.3	Standby → rotate	StatusClassifier, EndpointRotator	Classification
2.4	Sticky current primary	FailoverState	Failover loop
2.5	Redirect/standby/no-primary ≠ success	StatusClassifier	Classification
3.1, 4.1	No-primary → backoff to budget	XcecliRetryLoop, FailoverState	Failover loop
3.2	Success before budget	XcecliRetryLoop	Failover loop
3.3, 3.4	Budget exhausted → distinct cluster-not-ready	XcecliRetryLoop	Failover loop
4.2	Redirect → prompt switch, no backoff	EndpointRotator	Classification
4.4	Transport → reconnect/rotate	EndpointRotator, ReplyWaitWake	Classification
4.5	Non-retriable → surface	StatusClassifier	Classification
1.2	Re-establish session against new primary	ReconnectRejoinHook	Failover loop
5.1, 5.3	Reads+writes; no silent drop (reply-bearing)	XcecliRetryLoop, OpCategory	Failover loop
5.2	Write retry not double-applied	StatusClassifier (DUP→success) + controller idempotency	—
6.1, 6.2, 6.3	Configurable params, defaults, jitter	FailoverState, XcecliRetryLoop	Failover loop
7.1, 7.2	Log failover target + exhaustion	XcecliRetryLoop	—
8.1, 8.2	Controller populates redirect addr	ControllerRedirectAddress	—

Components and Interfaces

Component	Layer	Intent	Req Coverage	Key Dependencies	Contracts
XcecliRetryLoop	client (client.c)	do/while wrapper around each reply-bearing xce_* call	1.1,1.2,3.x,5.x,7.x	xcecli_retry_next(P0)	Service
xcecli_retry_next	client (client_failover.c)	Decision helper: classify, rotate, rejoin, backoff, budget	1.2,3.x,4.x,7.x	EndpointRotator(P0), StatusClassifier(P0), ReconnectRejoinHook(P0), FailoverState(P0)	Service
ReconnectRejoinHook	client (client.c)	Invoke consumer callback after reconnect, before resume	1.2	api_ctx_t(P0)	Service, State
OpCategory	client (client.c)	Per-op shape: reply-bearing / write / fire-and-forget	1.1,5.1,5.2	—	State
EndpointRotator	client (client_failover.c)	Reconnect to addr or next master over api_ctx	2.1,2.2,2.3,4.2,4.4	endpoint_connect(P0), FailoverState(P0)	Service
StatusClassifier	client (client_failover.c)	Map status → action	2.5,4.1,4.2,4.3,4.5,5.2	errnox codes(P0)	Service
FailoverState	client (api_ctx_t)	Master cursor, retry cfg, sticky primary	2.4,6.1	xce_client_config_t(P0)	State
RedirectCapture	ctx (xcectx.c)	Copy redirect-hint address into ctx (if present)	2.2,8.3	xce_reply_t(P0)	State
ReplyWaitWake	ctx (xcectx.c)	Release cbuf_get on endpoint error	4.4	endpoint err_handler(P0)	Service
ControllerRedirectAddress	controller	Fill redirect_host/port	8.1,8.2	ha_state primary_host/port(P0)	State

client (libxce)

XcecliRetryLoop — the wrapper pattern in each xcecli_*

Field	Detail
Intent	Turn each one-shot xce_* call into a transparent failover loop with no change to xce_*
Requirements	1.1, 1.2, 3.1, 3.2, 3.3, 3.4, 5.1, 5.3, 7.1, 7.2

Responsibilities & Constraints

• Each xcecli_* op runs its existing xce_* call inside a do/while driven by xcecli_retry_next.
• xce_* ops are unchanged: each is a single attempt that returns the controller status (fail fast).
• Never drops a request silently (5.3): the loop returns the op's success value or a distinct error.

Contracts: Service [x]

/* Pattern applied to every xcecli_* op (joblist shown; is_write=false for reads). */
int xcecli_joblist(xcecli_ctx_t c, xce_job_t ***out) {
    api_ctx_t      *ac = (api_ctx_t *)c;
    xcecli_retry_t  st = XCECLI_RETRY_INIT(ac);   /* attempt=0, delay/seed from ac->failover.retry */
    int rc;
    do {
        rc = xce_joblist(ac->ctx, out);           /* unchanged one-shot xceapi op */
    } while (xcecli_retry_next(ac, &rc, /*is_write=*/false, &st));
    return rc;                                     /* >=0 ok; -E_XCE_NOT_READY = cluster not ready */
}

• Postconditions: >=0 success (sticky primary updated, 2.4); -E_XCE_NOT_READY on no-primary budget exhaustion (3.3); terminal errors surfaced as-is (4.5). For writes, pass is_write=true.

xcecli_retry_next — decision helper

Field	Detail
Intent	Decide whether to retry after an xce_* returned rc; perform reconnect/rotate/backoff
Requirements	1.2, 3.1, 3.3, 3.4, 4.1, 4.2, 4.3, 4.4, 5.2, 7.1, 7.2

Responsibilities & Constraints

• Owns the retry budget, backoff progression, sticky-primary update, and cluster-not-ready mapping.
• After a transport error the existing path left ctx->state = XCE_CTX_ERR; on a successful reconnect this helper restores XCE_CTX_RUN so the next xce_* passes its guard (CI2).
• Drains any stale entry from ctx->wbuf after reconnect, before the next attempt (CI1); reconnect closes the old fd first, so the dead connection yields no further replies.

Contracts: Service [x]

typedef struct { int attempt; int delay_ms; unsigned int rand_seed; } xcecli_retry_t;

/* Returns true → caller re-invokes its xce_* op; false → stop, *rc is final.
 * On DUP write replay it sets *rc = 0 (success) and returns false (5.2). */
bool xcecli_retry_next(api_ctx_t *ac, int *rc, bool is_write, xcecli_retry_t *st);

• Decisions: SUCCESS/TERMINAL → return false; RETRY_NOW → EndpointRotator (hint-first) then true, no backoff (4.2); RETRY_BACKOFF → jittered sleep then EndpointRotator then true (4.1); attempts beyond retry.max_retries while no-primary → set *rc = -E_XCE_NOT_READY, return false (3.3, 3.4).
• Validation: INFO log target on rotation (7.1); WARN log attempt count on exhaustion (7.2).

EndpointRotator

Field	Detail
Intent	Re-point ac->ep to a redirect address or the next master in ac->conf
Requirements	2.1, 2.2, 2.3, 4.2, 4.4, 8.4

Responsibilities & Constraints

• On retry, if a redirect-address hint was captured, try it first (then clear it); otherwise advance the master cursor and reconnect to the next configured master (8.3, 8.4). A failed hint just falls through to rotation on the next retry — correctness never depends on it.
• Reuse the same endpoint_t (reply routing to ctx->wbuf preserved); close stale fd/ssl first.

Contracts: Service [x]

/* Reconnect ac->ep to the redirect-hint address if one is set (consumes it), else advance to the
 * next configured master in ac->conf and reconnect. Returns 0 or -errno. */
int xce_failover_rotate(api_ctx_t *ac);

StatusClassifier

Field	Detail
Intent	Map a reply status / transport rc to a failover action
Requirements	2.5, 4.1, 4.2, 4.3, 4.5, 5.2

Contracts: Service [x]

typedef enum {
    XCE_FO_SUCCESS,       /* status == 0, or idempotent-replay success (DUP on a re-sent write) */
    XCE_FO_RETRY_NOW,     /* E_XCE_REDIRECT / E_XCE_STANDBY / transport error — rotate, no backoff */
    XCE_FO_RETRY_BACKOFF, /* E_XCE_NOMASTER / E_XCE_NOT_READY — rotate after jittered backoff */
    XCE_FO_TERMINAL       /* AUTH/PERM/INVALID/NOENT and other non-retriable — surface */
} xce_fo_action_t;

/* attempt == 0 for the first send; >0 for a re-send after failover.
 * is_write distinguishes mutating ops (job submit) from reads.
 * The redirect ADDRESS is NOT a classifier concern — it is a hint consumed by EndpointRotator. */
xce_fo_action_t xce_failover_classify(int status, bool is_write, int attempt);

• Invariants: E_XCE_REDIRECT/STANDBY/NOMASTER/NOT_READY are never reported as success (2.5).
• Idempotent-replay rule (5.2): -E_XCE_DUP_JOB / -EEXIST on a re-sent write (is_write && attempt > 0) maps to XCE_FO_SUCCESS — the controller's client-supplied-jobid replay guard (job.c:56,95) means "already applied" = our write succeeded. On attempt == 0 the same status is a genuine duplicate and maps to XCE_FO_TERMINAL (surfaced unchanged).

OpCategory — per-op shape (CI1)

Field	Detail
Intent	Classify each xcecli_* op so its wrapper applies the right failover behavior
Requirements	1.1, 5.1, 5.2

Three categories drive the wrapper generated for each of the ~33 ops:

Category	Ops (examples)	xce_* shape	Failover behavior
Reply-read	joblist, reslist, workerlist, clusterconf, resval, hello, jobinfo	*_req + cbuf_get → status	Full failover; is_write=false
Reply-write	jobsubmit, jobctrl, queuectrl, hostctrl, update_resvals, runtask, join	*_req + cbuf_get → status	Full failover; is_write=true (DUP-replay→success)
Fire-and-forget	notify_* (jobstart/finish/fail/stop/chkpt), heartbeat, wakeup, nop	*_req only, no status	Best-effort: reconnect on a known-dead connection only; no standby/redirect detection, no no-primary backoff. Redelivery deferred to worker-rejoin-resync

• The exact per-op assignment is enumerated during task generation; join is reply-write and is the op that triggers E_XCE_REDIRECT at reconnect time (it carries the daemon's role/attrs).

ReconnectRejoinHook

Field	Detail
Intent	Let a stateful consumer re-establish its session on the new primary after a reconnect, before the interrupted op resumes
Requirements	1.2

Responsibilities & Constraints

• libxce owns the hook mechanism and ordering (reconnect → hook → resume); the callback content (join handshake, worker job re-sync) belongs to component-failover-adoption / worker-rejoin-resync.
• Invoked once per successful reconnect to a new endpoint, before the next xce_* attempt. If the hook fails (e.g. the new primary rejects the join), the failover loop treats it like a failed attempt (rotate/backoff per the returned status). Pure clients register no hook (no-op path).

Contracts: Service [x] / State [x]

/* Registered via the client API; cb runs on the calling thread after reconnect.
 * Return 0 to proceed with the interrupted op; <0 to treat the rejoin as failed. */
typedef int (*xcecli_reconnect_cb)(xcecli_ctx_t ctx, void *arg);
int xcecli_set_reconnect_handler(xcecli_ctx_t ctx, xcecli_reconnect_cb cb, void *arg);

client / ctx (libxce)

FailoverState (on api_ctx_t)

Contracts: State [x]

/* On api_ctx_t — persists across calls (sticky primary + tunables). */
typedef struct {
    mylock_t           lock;         /* serializes xcecli_* per ctx (CI2 — shared-ctx safety) */
    int                cur_master;   /* index into conf->masters[] (sticky primary) */
    xce_retry_config_t retry;        /* defaults from XCE_RETRY_DEFAULTS unless overridden */
    xcecli_reconnect_cb rejoin_cb;   /* consumer rejoin hook; NULL for pure clients */
    void               *rejoin_arg;
} xce_failover_state_t;

/* On xce_ctx_t — written by handle_reply (which only sees xce_ctx_t), read by the rotator via ac->ctx. */
/*   char     redirect_hint_host[XCE_MAXHOSTNAME_LEN];  hint from last redirect reply; "" if none      */
/*   uint16_t redirect_hint_port;                                                                  */

• State model: cur_master updated on a successful served-as-primary reply (2.4); retry config from XCE_RETRY_DEFAULTS unless overridden (6.1, 6.2). The redirect hint is read by the rotator on the next retry and cleared on consumption; it is purely advisory.
• Concurrency (CI2): the apiserver shares one ctx across goroutines (a heartbeat goroutine plus request handlers, clib.go:182). Each xcecli_* op takes failover.lock for the duration of its do/while loop, so reconnect/rotate (which closes ac->ep and drains wbuf) cannot race another in-flight call on the same ctx. The single-slot wbuf already implies one in-flight request per ctx; the lock makes that explicit and covers the reconnect window.

RedirectCapture (in handle_reply)

Contracts: State [x]

• On a redirect reply that carries an address, copy reply.redirect_host (capnp-owned, freed with the reply — see request.c:243-250 hazard) + redirect_port into the fixed-size xce_ctx_t hint fields before the reply is freed; leave the hint empty otherwise. Because the hint is advisory, the copy needs no sequence correlation or lifetime guarantees beyond "copy now, use-or-discard next retry." The status continues through the existing cbuf_tryput(&ctx->wbuf,&status) path (xcectx.c:303), so xceapi.c is unchanged.

ReplyWaitWake (endpoint error path)

Contracts: Service [x]

• On endpoint error/RDHUP for ctx->ep, release a pending cbuf_get with a transport error (-ECONNRESET/-ETIMEDOUT) so the xce_* call returns and the loop rotates instead of hanging (4.4, F7). The existing error path already injects an error into wbuf and sets ctx->state = XCE_CTX_ERR (xcectx.c:148-154) — this component reuses it.
• State reset (CI2): that error path leaves ctx->state = XCE_CTX_ERR, which the per-op guard rejects as -EINVAL (xceapi.c:14,38,…) — the reported bug. xcecli_retry_next restores ctx->state = XCE_CTX_RUN after a successful reconnect, before the next xce_* attempt, so the guard (re-checked per attempt) passes. CI1 drain: the helper also drains any stale wbuf entry after reconnect so a late reply from the dead connection is never consumed by the next attempt.

controller

ControllerRedirectAddress (join.c)

Contracts: State [x]

• On the E_XCE_REDIRECT path, set reply.redirect_host = c->ha.primary_host and reply.redirect_port = c->ha.primary_port (8.1). When the primary is unknown, keep returning E_XCE_NOMASTER rather than an addressless redirect (8.2 — already the case at join.c:270).
• Audit during implementation: confirm join.c is the sole E_XCE_REDIRECT emitter; apply the same populate to any other site found.

Error Handling

Error Strategy

Classify every round-trip outcome (StatusClassifier) into success / rotate-now / backoff / terminal, and bound the loop by the retry budget. Repair the context on transport failure rather than leaving it in a non-RUN state (F7) so failover is repeatable.

Context state model (CI2): the existing transport-error path sets ctx->state = XCE_CTX_ERR (xcectx.c:148); the per-op XCE_CTX_RUN guard then rejects with -EINVAL (xceapi.c:14,…) — the reported bug. xcecli_retry_next owns the transition back: after a successful reconnect it sets ctx->state = XCE_CTX_RUN before the next xce_* attempt, so the per-op guard (re-checked each attempt) passes. The guard otherwise stays exactly as today — xceapi.c is unchanged.

Stale-reply avoidance (CI1): ctx->wbuf is a single-slot, status-only channel (cbuf_init(...,1,sizeof(int)), xcectx.c:185). Reconnect closes the old fd first, so the dead connection produces no further replies; xcecli_retry_next then drains any entry already sitting in wbuf before the next attempt. This drain-after-close is sufficient — no per-reply sequence correlation is needed (a deliberate simplification).

Error Categories and Responses

• Transport (refused/timeout/reset): reconnect/rotate, retry; wake any pending wait (4.4).
• No primary (NOMASTER/NOT_READY): jittered backoff; on budget exhaustion return the distinct -E_XCE_NOT_READY cluster-not-ready code (3.3, 3.4) — never -EINVAL.
• Redirect/standby: rotate (address-first) without backoff (4.2, 4.3).
• Terminal (AUTH/PERM/INVALID/NOENT): surface immediately, unchanged (4.5).

Monitoring

INFO log on each rotation with the target endpoint (7.1); WARN log on budget exhaustion with the attempt count and last status (7.2). Reuse existing libxce logging.

Performance & Scalability

Rejoin-on-failover cost is accepted as-is (decision 2026-06-16): failover is a rare, exceptional event, so the cost is not on any hot path, and some handshake is unavoidable — a freshly elected primary has no live connection to a reconnecting daemon until it re-joins. Characteristics:

• Scheduler/apiserver join carry trivial payloads (NULL attrs); worker join carries a small hostattrs_t. All reuse the daemons' existing startup join path — no new heavy machinery.
• On rejoin, apiserver/scheduler receive full state (e.g. the full job list), not a delta; for a large cluster with many workers this is sizable but infrequent. This is simpler and correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF).
• Herd control: the jittered backoff (6.3) staggers simultaneous reconnects after a failover.

Deferred optimization (YAGNI): a lightweight "re-attach" (session-token hello letting the primary skip re-pushing already-replicated state, or delta-only state transfer) is intentionally not designed now — delta reconciliation is non-trivial and full-state-on-rejoin is acceptable. Revisit only if failover-rejoin latency is measured as a problem at scale. Recorded in roadmap.md backlog.

Testing Strategy

Unit Tests (test_libxce_retry.c, realigned)

• xce_failover_classify: REDIRECT+addr→ROTATE_ADDR; REDIRECT/STANDBY no-addr→ROTATE; NOMASTER/NOT_READY→BACKOFF; AUTH/PERM/INVALID/NOENT→TERMINAL; 0→SUCCESS (2.5, 4.1-4.5).
• xcecli_retry_next loop: NOMASTER until budget → returns -E_XCE_NOT_READY (3.3); becomes primary mid-loop → success, sticky cursor advanced (2.4, 3.2); non-retriable → immediate surface (4.5).
• Stale-reply avoidance (CI1): after a reconnect (old fd closed first), a stale entry left in wbuf is drained before the next attempt, so no prior-attempt status is consumed; no sequence matching (1.2, 5.3).
• Context state (CI2): after a transport error drives XCE_CTX_ERR, a successful reconnect restores XCE_CTX_RUN and the retry completes — never surfaces -EINVAL (1.2, 3.2).
• Idempotent write (CI3): DUP_JOB on a re-sent jobsub (attempt>0) → success; on attempt 0 → surfaced as duplicate (5.2).
• EndpointRotator: redirect address used first (8.3); absent → next master (8.4); rotation reuses same endpoint object.
• RetryConfig: NULL cfg → defaults (6.2); jitter staggers concurrent retries (6.3, reuse 50-client).

Integration / E2E (3-node, /xce-deploy-ha + xce_test)

• Submit jobs, kill primary, joblist: returns cluster-not-ready during the election window, then transparently succeeds against the new primary with no error (1.2, 3.2, 3.3) — the bug reproduction.
• Redirect path: contact a standby directly → client lands on primary in one hop via the populated redirect address (8.1, 8.3).
• Write failover: jobsub across a failover → exactly one job created (5.1, 5.2 via controller idempotency).

Controller Unit Test

• handle_join redirect path populates redirect_host/port from ha.primary_host/port (8.1); unknown primary still yields E_XCE_NOMASTER (8.2).