myxce/.kiro/specs/libxce-failover/design.md

# Design Document — libxce-failover

## Overview

**Purpose**: This feature delivers transparent controller-primary failover to every client
application built on libxce (worker, scheduler, apiserver via cgo, and the `xce_test` harness),
so that a controller failover is hidden when short and reported clearly when not.

**Users**: libxce callers issue normal requests (`xce_joblist`, `xce_jobsub`, …). When the
connected primary dies, libxce locates the newly elected primary and completes the request there,
or — if no primary exists yet — returns a distinct cluster-not-ready error instead of an opaque
`EINVAL`. No caller code changes.

**Impact**: Today the client connects to one master (`client.c:122`), routes every request through
the fire-and-forget `xce_request()`, and has no reply-status-driven retry on the production path
(`xce_request_with_retry()` only retries in test-hook mode). This design places failover at the
**`client.c` (`xcecli_*`) layer** — the universal client surface that every consumer goes through
(worker, scheduler, apiserver-via-cgo, `xce_test`) and the only layer whose context (`api_ctx_t`)
co-locates the master list (`conf`), the endpoint (`ep`), and the reply context (`ctx`). Each
`xcecli_*` op already calls its internal `xce_*` op, which **returns the controller status**; the
failover loop wraps that call, classifies the status, and reconnects/rotates over `api_ctx.conf`.
It adds master-list rotation, redirect-address following, and a small controller change so redirect
replies carry the primary address. `xceapi.c` ops are unchanged.

### Goals
- Route all libxce request/reply round trips through one transparent failover path (1.1, 1.3).
- Discover the new primary via redirect address, falling back to master-list rotation (2.x, 8.x).
- Hide short failovers via bounded backoff; surface a distinct cluster-not-ready on exhaustion (3.x).
- Fail over reads and writes uniformly, relying on controller idempotency for writes (5.x).
- Reuse the existing retry config/defaults; stagger retries with jitter (6.x).

### Non-Goals
- Controller-side election, WAL, commit barrier, write-idempotency logic (consumed, not changed).
- Per-daemon adoption (master-list wiring, liveness-safe blocking) — `component-failover-adoption`.
- New transport opcodes or new reply fields (redirect fields already exist).

## Boundary Commitments

### This Spec Owns
- The client-side failover loop: classify status → reconnect/rotate/backoff → resume, including the
  cluster-not-ready terminal mapping, for **reply-bearing** `xcecli_*` ops.
- Client-side master-list cursor and current-primary stickiness within a context.
- Capturing the redirect address from a reply as a best-effort hint (no correctness dependency).
- A **reconnect/rejoin hook**: a consumer-registered callback the failover machinery invokes after
  it reconnects to a new primary and *before* it resumes the interrupted op. libxce owns the hook
  mechanism, its ordering, and invoking it; it does **not** own the callback's content.
- A per-ctx concurrency lock so `xcecli_*` failover is safe when one ctx is shared across threads.
- The single controller change that populates `redirect_host/port` on `E_XCE_REDIRECT` replies.
- Realignment of the libxce retry unit tests to the real controller status taxonomy.

### Out of Boundary
- How the controller decides primary/standby/redirect/no-primary (election, `ha_should_redirect`).
- The **content** of the rejoin callback: the join/state-exchange a daemon performs on reconnect
  (owned by `component-failover-adoption`) and the worker's running-jobs re-sync (owned by
  `worker-rejoin-resync`). This spec provides and invokes the hook; the daemons fill it in.
- Redelivery of fire-and-forget notifications lost across a failover (also `worker-rejoin-resync`).
- Worker/scheduler/apiserver adoption and their threading/liveness handling of bounded blocking.
- Any reply payload beyond the already-defined `redirect_host/redirect_port`.

### Allowed Dependencies
- Existing controller status codes (`errnox.h:22-25`) and redirect reply fields (`typemap.h:225`).
- `endpoint_connect()` / endpoint I/O (`netio.*`), request encode/enqueue (`request.c`), and the
  context wait channel (`cbuf`, `xcectx.c`).
- `xce_retry_config_t` + `XCE_RETRY_DEFAULTS` (request.h) and the cluster/peer config (`cluster.*`).

### Revalidation Triggers
- Change to the reply `status`/`redirect_*` contract → re-check client classification + controller.
- Change to the async reply-dispatch model (`handle_reply` → `cbuf` signalling) → re-check round trip.
- Change to `api_ctx_t`/endpoint ownership → re-check reconnect-reuses-same-endpoint assumption.
- New non-primary status code added to the controller → extend client classification.
- Change to the reconnect/rejoin hook signature or its invocation ordering → re-check daemon rejoin
  registration (`component-failover-adoption`) and worker resync (`worker-rejoin-resync`).

## Architecture

### Existing Architecture Analysis
- **Layering — `xcecli_*` is the universal surface.** Every consumer calls `client.c`'s `xcecli_*`
  ops, never `xceapi.c`'s `xce_*` directly: worker (`controller_connector.c:184,272`), scheduler
  (`scheduler.c:457,463`), apiserver via cgo (`clib.go:34,36`), `xce_test`. Each `xcecli_*` is a thin
  wrapper that calls its `xce_*` counterpart, which **returns the controller status** as its rc.
- **Only `api_ctx_t` reaches the master list.** `api_ctx_t = { conf (masters), ep, ctx }`
  (client.c:23). The inner `xce_ctx_t` (xceapi) holds `ep`/`wbuf`/`state` but **no** `conf`
  (xcectx.h) — so failover belongs at the client layer, where the master list is already in scope.
- **`xce_*` fail fast; `xcecli_*` retries.** The async send/wait stays inside `xce_*` unchanged: a
  single attempt that returns the controller status (or transport error). The `xcecli_*` wrapper
  classifies that rc and, on a retriable status, reconnects/rotates `api_ctx.ep` over `api_ctx.conf`
  and re-invokes `xce_*`. No async internals, thunks, or master-list plumbing are introduced.
- **Reconnect reuses the endpoint.** `endpoint_connect(ep,host,port)` re-points the same
  `endpoint_t`, preserving the `msg_handler` wiring that routes replies to `ctx->wbuf`.
- **Stateful consumers need a rejoin on reconnect.** `xce_test` is a pure client — reconnect +
  resend suffices. But the apiserver, worker, and scheduler hold a *session* with the primary
  established via `xcecli_join` (role + attrs); a freshly elected primary has no live session for a
  reconnecting daemon until it re-joins, and a worker must additionally re-sync its running-jobs
  state. So failover for these consumers is **reconnect → rejoin/state-exchange → resume**, not just
  reconnect → resend. libxce models this with a consumer-registered reconnect hook; the hook content
  lives in `component-failover-adoption` (join) and `worker-rejoin-resync` (job state).
- **Two op shapes.** Reply-bearing ops (`xce_joblist`/`xce_jobsubmit`/`xce_hello`/…) do
  `xce_*_req` + `cbuf_get` → return a controller status the loop can classify. Fire-and-forget ops
  (`xce_notify*`, heartbeat) do `xce_*_req` only (no `cbuf_get`, xceapi.c:282) → return the enqueue
  result, never a controller status. Failover classification applies only to the former.
- **Patterns preserved**: `xce_*` ops, `xce_*_req` enqueue wrappers, and `xce_request`/`xce_reply`
  are all unchanged; failover is additive in `xcecli_*`. Trade-off: ~33 `xcecli_*` wrappers each gain
  a small retry loop (more lines), but the policy lives in one shared decision helper (fewer moving
  parts) — an explicit simplicity-over-LOC choice.

### Architecture Pattern & Boundary Map

```mermaid
graph LR
    Caller[consumer worker scheduler apiserver xce_test] --> Cli[xcecli op in client.c]
    Cli --> Loop[do while retry loop]
    Loop --> Decide[xcecli_retry_next decision helper]
    Loop --> Xce[xce op in xceapi one shot]
    Xce --> Ctrl[controller]
    Ctrl --> Reply[handle_reply]
    Reply --> Capture[capture redirect hint into ctx]
    Decide --> Rotate[reconnect or rotate over api_ctx conf and ep]
    Decide --> Backoff[jittered backoff]
```

- **Selected pattern**: a thin retry loop in each `xcecli_*` wrapper, driven by one shared decision
  helper (`xcecli_retry_next`). Rationale: `client.c` is the universal surface AND the only layer
  whose `api_ctx_t` co-locates master list + endpoint + ctx; `xce_*` already returns the status.
- **Dependency direction**: `types → cluster/config → endpoint(netio) → xceapi(xce_*) → client(xcecli_*)`.
  Each layer imports only leftward. Failover logic is added at the `client` layer (rightmost),
  depending on `xce_*` and `cluster/config`; `xceapi` does not depend on it.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Client wrappers | C (gnu17), libxce `client.c` | `xcecli_*` retry loop + decision helper + rotate | Reuses `xce_*`, `endpoint_connect`, `cluster` |
| Client API | C (gnu17), libxce `xceapi.c` | **Unchanged** — one-shot `xce_*` returns status | No edits |
| Controller | C (gnu17) | Populate redirect address on `E_XCE_REDIRECT` | ~2 lines in `join.c`; fields already in schema |
| Tests | GoogleTest (testsuite) | Realign retry tests to real status taxonomy | Test the `xcecli_*` retry path |

## File Structure Plan

### New Files
```
src/lib/client_failover.c       # xcecli_retry_next decision helper, StatusClassifier, rotate/reconnect over api_ctx
src/lib/client_internal.h        # private: api_ctx_t + xce_failover_state_t shared by client.c and client_failover.c
```
> `api_ctx_t` is currently file-local to `client.c`; move its definition to `client_internal.h` so the
> failover helper can operate on it without exposing it in the public API.

### Modified Files
- `src/lib/client.c` — wrap each **reply-bearing** `xcecli_*` op body in a `do { rc =
  xce_<op>(...); } while (xcecli_retry_next(ac,&rc,is_write,&st));` loop (per OpCategory);
  fire-and-forget ops get the transport-reconnect-only variant; add `xce_failover_state_t` to
  `api_ctx_t` (lock + cursor + retry config + rejoin hook); add `xcecli_set_reconnect_handler()`;
  init + take `failover.lock` per op; init in `xcecli_start()`; keep initial connect to `masters[0]`.
- `src/include/client.h` — declare `xcecli_set_reconnect_handler()` + `xcecli_reconnect_cb`.
- `src/lib/xcectx.c` — `handle_reply()` copies the redirect-hint address (`reply.redirect_host/port`,
  when present) into `xce_ctx_t` before the reply is freed; the existing error path (`XCE_CTX_ERR` +
  `wbuf` error inject, xcectx.c:137-154) is the wake hook reused on reconnect.
- `src/include/xcectx.h` — add the fixed-size redirect-hint `host/port` fields to `xce_ctx_t`.
- `src/include/request.h` / `src/lib/request.c` — keep `xce_retry_config_t` + `XCE_RETRY_DEFAULTS`;
  **remove** the orphaned no-op `xce_request_with_retry()` (superseded by the `xcecli_*` loop).
- `src/lib/xceapi.c` — **unchanged** (one-shot `xce_*` ops already return the controller status).
- `src/controller/join.c` — populate `reply.redirect_host = c->ha.primary_host` /
  `reply.redirect_port = c->ha.primary_port` on the `E_XCE_REDIRECT` path (audit for other sites).
- `src/lib/testsuite/test_libxce_retry.c` — retarget from the retired primitive to the `xcecli_*`
  retry path + `StatusClassifier`; cover `E_XCE_REDIRECT`/`NOMASTER`/`NOT_READY`/`STANDBY`,
  master rotation, and budget→not-ready.

## System Flows

### Failover loop (per `xcecli_*` op)

```mermaid
sequenceDiagram
    participant C as Caller
    participant F as xcecli op do while loop
    participant E as Endpoint
    participant K as Controller
    C->>F: xcecli op
    loop until success, terminal, or budget exhausted
        F->>E: xce op one shot
        E->>K: request
        K-->>E: reply status plus optional redirect addr
        E-->>F: xce op returns status
        alt status success
            F-->>C: ok
        else redirect or standby or transport error
            F->>E: rotate (hint address first if set, else next master) then invoke rejoin hook
        else no-primary or not-ready
            F->>F: backoff jittered then rotate
        else non-retriable
            F-->>C: surface error
        end
    end
    F-->>C: cluster-not-ready on budget exhaustion
```

After any reconnect to a new endpoint, the failover machinery invokes the consumer-registered
reconnect/rejoin hook (if set) **before** re-sending the interrupted op, so a stateful daemon
re-establishes its session (and a worker re-syncs job state) against the new primary. Pure clients
(`xce_test`) register no hook and skip straight to re-send.

Key decisions: redirect/standby/transport rotate **without** the no-primary backoff delay (4.2);
only no-primary/not-ready consume the backoff budget (3.1, 4.1); on success the served master
becomes the sticky current primary (2.4).

### Status classification

```mermaid
stateDiagram-v2
    [*] --> Classify
    Classify --> Success: status==0 or DUP on re-sent write
    Classify --> RetryNow: REDIRECT or STANDBY or transport error
    Classify --> RetryBackoff: NOMASTER or NOT_READY
    Classify --> Terminal: AUTH PERM INVALID NOENT
    RetryNow --> Rotate
    RetryBackoff --> Rotate
    Rotate --> Classify: hint first else next master
    Success --> [*]
    Terminal --> [*]
```

## Requirements Traceability

| Requirement | Summary | Components | Flows |
|-------------|---------|------------|-------|
| 1.1, 1.3 | All ops through failover; preserve API | XcecliRetryLoop (all `xcecli_*`) | Failover loop |
| 1.2 | Complete next request against new primary | XcecliRetryLoop, EndpointRotator | Failover loop |
| 2.1 | Unreachable → try other masters | EndpointRotator | Classification |
| 2.2, 8.3, 8.4 | Redirect addr if present else rotate | RedirectCapture, EndpointRotator | Classification |
| 2.3, 4.3 | Standby → rotate | StatusClassifier, EndpointRotator | Classification |
| 2.4 | Sticky current primary | FailoverState | Failover loop |
| 2.5 | Redirect/standby/no-primary ≠ success | StatusClassifier | Classification |
| 3.1, 4.1 | No-primary → backoff to budget | XcecliRetryLoop, FailoverState | Failover loop |
| 3.2 | Success before budget | XcecliRetryLoop | Failover loop |
| 3.3, 3.4 | Budget exhausted → distinct cluster-not-ready | XcecliRetryLoop | Failover loop |
| 4.2 | Redirect → prompt switch, no backoff | EndpointRotator | Classification |
| 4.4 | Transport → reconnect/rotate | EndpointRotator, ReplyWaitWake | Classification |
| 4.5 | Non-retriable → surface | StatusClassifier | Classification |
| 1.2 | Re-establish session against new primary | ReconnectRejoinHook | Failover loop |
| 5.1, 5.3 | Reads+writes; no silent drop (reply-bearing) | XcecliRetryLoop, OpCategory | Failover loop |
| 5.2 | Write retry not double-applied | StatusClassifier (DUP→success) + controller idempotency | — |
| 6.1, 6.2, 6.3 | Configurable params, defaults, jitter | FailoverState, XcecliRetryLoop | Failover loop |
| 7.1, 7.2 | Log failover target + exhaustion | XcecliRetryLoop | — |
| 8.1, 8.2 | Controller populates redirect addr | ControllerRedirectAddress | — |

## Components and Interfaces

| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|-------|--------|--------------|------------------|-----------|
| XcecliRetryLoop | client (`client.c`) | `do/while` wrapper around each reply-bearing `xce_*` call | 1.1,1.2,3.x,5.x,7.x | xcecli_retry_next(P0) | Service |
| xcecli_retry_next | client (`client_failover.c`) | Decision helper: classify, rotate, rejoin, backoff, budget | 1.2,3.x,4.x,7.x | EndpointRotator(P0), StatusClassifier(P0), ReconnectRejoinHook(P0), FailoverState(P0) | Service |
| ReconnectRejoinHook | client (`client.c`) | Invoke consumer callback after reconnect, before resume | 1.2 | api_ctx_t(P0) | Service, State |
| OpCategory | client (`client.c`) | Per-op shape: reply-bearing / write / fire-and-forget | 1.1,5.1,5.2 | — | State |
| EndpointRotator | client (`client_failover.c`) | Reconnect to addr or next master over api_ctx | 2.1,2.2,2.3,4.2,4.4 | endpoint_connect(P0), FailoverState(P0) | Service |
| StatusClassifier | client (`client_failover.c`) | Map status → action | 2.5,4.1,4.2,4.3,4.5,5.2 | errnox codes(P0) | Service |
| FailoverState | client (`api_ctx_t`) | Master cursor, retry cfg, sticky primary | 2.4,6.1 | xce_client_config_t(P0) | State |
| RedirectCapture | ctx (`xcectx.c`) | Copy redirect-hint address into ctx (if present) | 2.2,8.3 | xce_reply_t(P0) | State |
| ReplyWaitWake | ctx (`xcectx.c`) | Release cbuf_get on endpoint error | 4.4 | endpoint err_handler(P0) | Service |
| ControllerRedirectAddress | controller | Fill redirect_host/port | 8.1,8.2 | ha_state primary_host/port(P0) | State |

### client (libxce)

#### XcecliRetryLoop — the wrapper pattern in each `xcecli_*`

| Field | Detail |
|-------|--------|
| Intent | Turn each one-shot `xce_*` call into a transparent failover loop with no change to `xce_*` |
| Requirements | 1.1, 1.2, 3.1, 3.2, 3.3, 3.4, 5.1, 5.3, 7.1, 7.2 |

**Responsibilities & Constraints**
- Each `xcecli_*` op runs its existing `xce_*` call inside a `do/while` driven by `xcecli_retry_next`.
- `xce_*` ops are unchanged: each is a single attempt that returns the controller status (fail fast).
- Never drops a request silently (5.3): the loop returns the op's success value or a distinct error.

**Contracts**: Service [x]
```c
/* Pattern applied to every xcecli_* op (joblist shown; is_write=false for reads). */
int xcecli_joblist(xcecli_ctx_t c, xce_job_t ***out) {
    api_ctx_t      *ac = (api_ctx_t *)c;
    xcecli_retry_t  st = XCECLI_RETRY_INIT(ac);   /* attempt=0, delay/seed from ac->failover.retry */
    int rc;
    do {
        rc = xce_joblist(ac->ctx, out);           /* unchanged one-shot xceapi op */
    } while (xcecli_retry_next(ac, &rc, /*is_write=*/false, &st));
    return rc;                                     /* >=0 ok; -E_XCE_NOT_READY = cluster not ready */
}
```
- Postconditions: `>=0` success (sticky primary updated, 2.4); `-E_XCE_NOT_READY` on no-primary
  budget exhaustion (3.3); terminal errors surfaced as-is (4.5). For writes, pass `is_write=true`.

#### xcecli_retry_next — decision helper

| Field | Detail |
|-------|--------|
| Intent | Decide whether to retry after an `xce_*` returned `rc`; perform reconnect/rotate/backoff |
| Requirements | 1.2, 3.1, 3.3, 3.4, 4.1, 4.2, 4.3, 4.4, 5.2, 7.1, 7.2 |

**Responsibilities & Constraints**
- Owns the retry budget, backoff progression, sticky-primary update, and cluster-not-ready mapping.
- After a transport error the existing path left `ctx->state = XCE_CTX_ERR`; on a successful reconnect
  this helper restores `XCE_CTX_RUN` so the next `xce_*` passes its guard (CI2).
- Drains any stale entry from `ctx->wbuf` after reconnect, before the next attempt (CI1); reconnect
  closes the old fd first, so the dead connection yields no further replies.

**Contracts**: Service [x]
```c
typedef struct { int attempt; int delay_ms; unsigned int rand_seed; } xcecli_retry_t;

/* Returns true → caller re-invokes its xce_* op; false → stop, *rc is final.
 * On DUP write replay it sets *rc = 0 (success) and returns false (5.2). */
bool xcecli_retry_next(api_ctx_t *ac, int *rc, bool is_write, xcecli_retry_t *st);
```
- Decisions: SUCCESS/TERMINAL → return false; RETRY_NOW → `EndpointRotator` (hint-first) then true,
  no backoff (4.2); RETRY_BACKOFF → jittered sleep then `EndpointRotator` then true (4.1); attempts
  beyond `retry.max_retries` while no-primary → set `*rc = -E_XCE_NOT_READY`, return false (3.3, 3.4).
- Validation: INFO log target on rotation (7.1); WARN log attempt count on exhaustion (7.2).

#### EndpointRotator

| Field | Detail |
|-------|--------|
| Intent | Re-point `ac->ep` to a redirect address or the next master in `ac->conf` |
| Requirements | 2.1, 2.2, 2.3, 4.2, 4.4, 8.4 |

**Responsibilities & Constraints**
- On retry, if a redirect-address **hint** was captured, try it first (then clear it); otherwise
  advance the master cursor and reconnect to the next configured master (8.3, 8.4). A failed hint
  just falls through to rotation on the next retry — correctness never depends on it.
- Reuse the same `endpoint_t` (reply routing to `ctx->wbuf` preserved); close stale fd/ssl first.

**Contracts**: Service [x]
```c
/* Reconnect ac->ep to the redirect-hint address if one is set (consumes it), else advance to the
 * next configured master in ac->conf and reconnect. Returns 0 or -errno. */
int xce_failover_rotate(api_ctx_t *ac);
```

#### StatusClassifier

| Field | Detail |
|-------|--------|
| Intent | Map a reply status / transport rc to a failover action |
| Requirements | 2.5, 4.1, 4.2, 4.3, 4.5, 5.2 |

**Contracts**: Service [x]
```c
typedef enum {
    XCE_FO_SUCCESS,       /* status == 0, or idempotent-replay success (DUP on a re-sent write) */
    XCE_FO_RETRY_NOW,     /* E_XCE_REDIRECT / E_XCE_STANDBY / transport error — rotate, no backoff */
    XCE_FO_RETRY_BACKOFF, /* E_XCE_NOMASTER / E_XCE_NOT_READY — rotate after jittered backoff */
    XCE_FO_TERMINAL       /* AUTH/PERM/INVALID/NOENT and other non-retriable — surface */
} xce_fo_action_t;

/* attempt == 0 for the first send; >0 for a re-send after failover.
 * is_write distinguishes mutating ops (job submit) from reads.
 * The redirect ADDRESS is NOT a classifier concern — it is a hint consumed by EndpointRotator. */
xce_fo_action_t xce_failover_classify(int status, bool is_write, int attempt);
```
- Invariants: `E_XCE_REDIRECT`/`STANDBY`/`NOMASTER`/`NOT_READY` are never reported as success (2.5).
- **Idempotent-replay rule (5.2):** `-E_XCE_DUP_JOB` / `-EEXIST` on a **re-sent write** (`is_write &&
  attempt > 0`) maps to `XCE_FO_SUCCESS` — the controller's client-supplied-`jobid` replay guard
  (`job.c:56,95`) means "already applied" = our write succeeded. On `attempt == 0` the same status
  is a genuine duplicate and maps to `XCE_FO_TERMINAL` (surfaced unchanged).

#### OpCategory — per-op shape (CI1)

| Field | Detail |
|-------|--------|
| Intent | Classify each `xcecli_*` op so its wrapper applies the right failover behavior |
| Requirements | 1.1, 5.1, 5.2 |

Three categories drive the wrapper generated for each of the ~33 ops:

| Category | Ops (examples) | `xce_*` shape | Failover behavior |
|----------|----------------|---------------|-------------------|
| Reply-read | `joblist`, `reslist`, `workerlist`, `clusterconf`, `resval`, `hello`, `jobinfo` | `*_req` + `cbuf_get` → status | Full failover; `is_write=false` |
| Reply-write | `jobsubmit`, `jobctrl`, `queuectrl`, `hostctrl`, `update_resvals`, `runtask`, `join` | `*_req` + `cbuf_get` → status | Full failover; `is_write=true` (DUP-replay→success) |
| Fire-and-forget | `notify_*` (jobstart/finish/fail/stop/chkpt), `heartbeat`, `wakeup`, `nop` | `*_req` only, **no status** | Best-effort: reconnect on a known-dead connection only; **no** standby/redirect detection, **no** no-primary backoff. Redelivery deferred to `worker-rejoin-resync` |

- The exact per-op assignment is enumerated during task generation; `join` is reply-write and is the
  op that triggers `E_XCE_REDIRECT` at reconnect time (it carries the daemon's role/attrs).

#### ReconnectRejoinHook

| Field | Detail |
|-------|--------|
| Intent | Let a stateful consumer re-establish its session on the new primary after a reconnect, before the interrupted op resumes |
| Requirements | 1.2 |

**Responsibilities & Constraints**
- libxce owns the hook *mechanism* and *ordering* (reconnect → hook → resume); the callback *content*
  (join handshake, worker job re-sync) belongs to `component-failover-adoption` / `worker-rejoin-resync`.
- Invoked once per successful reconnect to a *new* endpoint, before the next `xce_*` attempt. If the
  hook fails (e.g. the new primary rejects the join), the failover loop treats it like a failed
  attempt (rotate/backoff per the returned status). Pure clients register no hook (no-op path).

**Contracts**: Service [x] / State [x]
```c
/* Registered via the client API; cb runs on the calling thread after reconnect.
 * Return 0 to proceed with the interrupted op; <0 to treat the rejoin as failed. */
typedef int (*xcecli_reconnect_cb)(xcecli_ctx_t ctx, void *arg);
int xcecli_set_reconnect_handler(xcecli_ctx_t ctx, xcecli_reconnect_cb cb, void *arg);
```

### client / ctx (libxce)

#### FailoverState (on `api_ctx_t`)

**Contracts**: State [x]
```c
/* On api_ctx_t — persists across calls (sticky primary + tunables). */
typedef struct {
    mylock_t           lock;         /* serializes xcecli_* per ctx (CI2 — shared-ctx safety) */
    int                cur_master;   /* index into conf->masters[] (sticky primary) */
    xce_retry_config_t retry;        /* defaults from XCE_RETRY_DEFAULTS unless overridden */
    xcecli_reconnect_cb rejoin_cb;   /* consumer rejoin hook; NULL for pure clients */
    void               *rejoin_arg;
} xce_failover_state_t;

/* On xce_ctx_t — written by handle_reply (which only sees xce_ctx_t), read by the rotator via ac->ctx. */
/*   char     redirect_hint_host[XCE_MAXHOSTNAME_LEN];  hint from last redirect reply; "" if none      */
/*   uint16_t redirect_hint_port;                                                                  */
```
- State model: `cur_master` updated on a successful served-as-primary reply (2.4); retry config from
  `XCE_RETRY_DEFAULTS` unless overridden (6.1, 6.2). The redirect hint is read by the rotator on the
  next retry and cleared on consumption; it is purely advisory.
- **Concurrency (CI2):** the apiserver shares one ctx across goroutines (a heartbeat goroutine plus
  request handlers, `clib.go:182`). Each `xcecli_*` op takes `failover.lock` for the duration of its
  `do/while` loop, so reconnect/rotate (which closes `ac->ep` and drains `wbuf`) cannot race another
  in-flight call on the same ctx. The single-slot `wbuf` already implies one in-flight request per
  ctx; the lock makes that explicit and covers the reconnect window.

#### RedirectCapture (in `handle_reply`)

**Contracts**: State [x]
- On a redirect reply that carries an address, copy `reply.redirect_host` (capnp-owned, freed with
  the reply — see request.c:243-250 hazard) + `redirect_port` into the fixed-size `xce_ctx_t` hint
  fields before the reply is freed; leave the hint empty otherwise. Because the hint is advisory, the
  copy needs no sequence correlation or lifetime guarantees beyond "copy now, use-or-discard next
  retry." The status continues through the existing `cbuf_tryput(&ctx->wbuf,&status)` path
  (xcectx.c:303), so `xceapi.c` is unchanged.

#### ReplyWaitWake (endpoint error path)

**Contracts**: Service [x]
- On endpoint error/RDHUP for `ctx->ep`, release a pending `cbuf_get` with a transport error
  (`-ECONNRESET`/`-ETIMEDOUT`) so the `xce_*` call returns and the loop rotates instead of hanging
  (4.4, F7). The existing error path already injects an error into `wbuf` and sets `ctx->state =
  XCE_CTX_ERR` (xcectx.c:148-154) — this component reuses it.
- **State reset (CI2):** that error path leaves `ctx->state = XCE_CTX_ERR`, which the per-op guard
  rejects as `-EINVAL` (xceapi.c:14,38,…) — the reported bug. `xcecli_retry_next` restores
  `ctx->state = XCE_CTX_RUN` after a successful reconnect, before the next `xce_*` attempt, so the
  guard (re-checked per attempt) passes. **CI1 drain:** the helper also drains any stale `wbuf` entry
  after reconnect so a late reply from the dead connection is never consumed by the next attempt.

### controller

#### ControllerRedirectAddress (`join.c`)

**Contracts**: State [x]
- On the `E_XCE_REDIRECT` path, set `reply.redirect_host = c->ha.primary_host` and
  `reply.redirect_port = c->ha.primary_port` (8.1). When the primary is unknown, keep returning
  `E_XCE_NOMASTER` rather than an addressless redirect (8.2 — already the case at join.c:270).
- Audit during implementation: confirm `join.c` is the sole `E_XCE_REDIRECT` emitter; apply the
  same populate to any other site found.

## Error Handling

### Error Strategy
Classify every round-trip outcome (StatusClassifier) into success / rotate-now / backoff / terminal,
and bound the loop by the retry budget. Repair the context on transport failure rather than leaving
it in a non-RUN state (F7) so failover is repeatable.

**Context state model (CI2):** the existing transport-error path sets `ctx->state = XCE_CTX_ERR`
(xcectx.c:148); the per-op `XCE_CTX_RUN` guard then rejects with `-EINVAL` (xceapi.c:14,…) — the
reported bug. `xcecli_retry_next` owns the transition back: after a successful reconnect it sets
`ctx->state = XCE_CTX_RUN` before the next `xce_*` attempt, so the per-op guard (re-checked each
attempt) passes. The guard otherwise stays exactly as today — `xceapi.c` is unchanged.

**Stale-reply avoidance (CI1):** `ctx->wbuf` is a single-slot, status-only channel
(`cbuf_init(...,1,sizeof(int))`, xcectx.c:185). Reconnect closes the old fd **first**, so the dead
connection produces no further replies; `xcecli_retry_next` then drains any entry already sitting in
`wbuf` before the next attempt. This drain-after-close is sufficient — no per-reply sequence
correlation is needed (a deliberate simplification).

### Error Categories and Responses
- **Transport (refused/timeout/reset)**: reconnect/rotate, retry; wake any pending wait (4.4).
- **No primary (NOMASTER/NOT_READY)**: jittered backoff; on budget exhaustion return the distinct
  `-E_XCE_NOT_READY` cluster-not-ready code (3.3, 3.4) — never `-EINVAL`.
- **Redirect/standby**: rotate (address-first) without backoff (4.2, 4.3).
- **Terminal (AUTH/PERM/INVALID/NOENT)**: surface immediately, unchanged (4.5).

### Monitoring
INFO log on each rotation with the target endpoint (7.1); WARN log on budget exhaustion with the
attempt count and last status (7.2). Reuse existing libxce logging.

## Performance & Scalability

Rejoin-on-failover cost is **accepted as-is** (decision 2026-06-16): failover is a rare, exceptional
event, so the cost is not on any hot path, and *some* handshake is unavoidable — a freshly elected
primary has no live connection to a reconnecting daemon until it re-joins. Characteristics:

- Scheduler/apiserver join carry trivial payloads (`NULL` attrs); worker join carries a small
  `hostattrs_t`. All reuse the daemons' existing startup join path — no new heavy machinery.
- On rejoin, apiserver/scheduler receive **full** state (e.g. the full job list), not a delta; for a
  large cluster with many workers this is sizable but infrequent. This is simpler and correct, and
  still cheaper than a full-history replace with no snapshot (cf. IBM LSF).
- **Herd control**: the jittered backoff (6.3) staggers simultaneous reconnects after a failover.

**Deferred optimization (YAGNI):** a lightweight "re-attach" (session-token hello letting the primary
skip re-pushing already-replicated state, or delta-only state transfer) is intentionally **not**
designed now — delta reconciliation is non-trivial and full-state-on-rejoin is acceptable. Revisit
only if failover-rejoin latency is *measured* as a problem at scale. Recorded in `roadmap.md` backlog.

## Testing Strategy

### Unit Tests (`test_libxce_retry.c`, realigned)
- `xce_failover_classify`: REDIRECT+addr→ROTATE_ADDR; REDIRECT/STANDBY no-addr→ROTATE;
  NOMASTER/NOT_READY→BACKOFF; AUTH/PERM/INVALID/NOENT→TERMINAL; 0→SUCCESS (2.5, 4.1-4.5).
- `xcecli_retry_next` loop: NOMASTER until budget → returns `-E_XCE_NOT_READY` (3.3); becomes primary
  mid-loop → success, sticky cursor advanced (2.4, 3.2); non-retriable → immediate surface (4.5).
- Stale-reply avoidance (CI1): after a reconnect (old fd closed first), a stale entry left in `wbuf`
  is drained before the next attempt, so no prior-attempt status is consumed; no sequence matching (1.2, 5.3).
- Context state (CI2): after a transport error drives `XCE_CTX_ERR`, a successful reconnect restores
  `XCE_CTX_RUN` and the retry completes — never surfaces `-EINVAL` (1.2, 3.2).
- Idempotent write (CI3): `DUP_JOB` on a re-sent `jobsub` (attempt>0) → success; on attempt 0 →
  surfaced as duplicate (5.2).
- EndpointRotator: redirect address used first (8.3); absent → next master (8.4); rotation reuses
  same endpoint object.
- RetryConfig: NULL cfg → defaults (6.2); jitter staggers concurrent retries (6.3, reuse 50-client).

### Integration / E2E (3-node, `/xce-deploy-ha` + `xce_test`)
- Submit jobs, kill primary, `joblist`: returns cluster-not-ready during the election window, then
  transparently succeeds against the new primary with no error (1.2, 3.2, 3.3) — the bug reproduction.
- Redirect path: contact a standby directly → client lands on primary in one hop via the populated
  redirect address (8.1, 8.3).
- Write failover: `jobsub` across a failover → exactly one job created (5.1, 5.2 via controller idempotency).

### Controller Unit Test
- `handle_join` redirect path populates `redirect_host/port` from `ha.primary_host/port` (8.1);
  unknown primary still yields `E_XCE_NOMASTER` (8.2).