# Research Log — libxce-failover

## Discovery Scope

Light, integration-focused discovery (extension of existing libxce). Goal: find the exact
seams to wire transparent failover into the client request/reply path. Verified by reading the
endpoint, client-config, request-core, reply, and controller-redirect code paths.

## Key Findings

### F1 — The async request/reply model determines the chokepoint (DECISIVE)
`xce_request(ep, req)` (request.c:317) only **encodes + enqueues** a packet — fire-and-forget.
The controller's reply (and its `status`) is delivered asynchronously: the endpoint's
`msg_handler` runs `handle_reply()` (xcectx.c:30) which signals the caller's wait channel, and
public ops block on `cbuf_get(&ctx->wbuf, &status)` (xceapi.c:116). Therefore the controller
status (`E_XCE_REDIRECT`/`NOMASTER`/`NOT_READY`/`STANDBY`) is observable only at the
**send-and-wait round trip**, not at `xce_request`. Failover orchestration must wrap that round
trip, where the `api_ctx_t` is in scope.

### F2 — `xce_request_with_retry()` production branch is a no-op stub
request.c:1972-1981: with no test hook installed, it does `xce_request(ep,req); return 0;`. The
entire retry/classify/backoff loop executes only under `xce_retry_send_hook` (tests). So the
"done" primitive cannot fail over against a real controller. Its hook contract
(`int hook(ep, req, reply_out)`) fakes a synchronous reply that production never produces.

### F3 — Redirect address is plumbed in the wire/struct but never populated or surfaced
- Wire schema has the fields: `controller.capnp:568-569` (`redirectHost`/`redirectPort`).
- Reply struct has them: `typemap.h:225-226` (`char *redirect_host; uint16_t redirect_port;`).
- Controller never sets them: every redirect site (`join.c:264`) sends `E_XCE_REDIRECT` with
  `data_which = nodata`; grep finds **zero** production assignments to `reply.redirect_*`.
- Client never surfaces them: `handle_reply()` (xcectx.c:34) returns only `reply.status`; the
  redirect fields are dropped before any caller sees them.

### F4 — Context already holds everything failover needs
`api_ctx_t` (client.c:23): `{ xce_client_config_t *conf; endpoint_t ep; xce_ctx_t ctx; }`.
`xce_client_config_t` (client.h:16): `masters[N_MAX_MASTERS]` + `master_ports[]` + `nmasters`.
So the master list, the endpoint to reconnect, and the reply channel are all reachable from one
context object. Missing: a current-master cursor, the retry config, and captured redirect target.

### F5 — Endpoint reconnect is straightforward and reusable
`endpoint_connect(ep, host, port)` (netio.c:659) does socket connect + SSL handshake on the
**same** `endpoint_t`. Reconnecting to a different master reuses the same endpoint object, so the
`msg_handler`/`owner` wiring that routes replies to `ctx->wbuf` stays intact.

### F6 — `xcecli_start()` only ever connects `masters[0]` (client.c:122)
No primary discovery at startup; single endpoint, never re-selected. Confirms the gap.

### F7 — Original EINVAL source
`xce_get_with_condition()` returns `-EINVAL` when `ctx->state != XCE_CTX_RUN` (xceapi.c:104).
After the primary dies the connection breaks and the ctx is left non-RUN, so every subsequent
call short-circuits to `-EINVAL` — the opaque error in the bug report. Failover must repair the
endpoint/ctx state instead of leaving it dead.

## Decisions (synthesis)

- **D1 (final — client-layer, supersedes the earlier xceapi round-trip idea):** failover lives in
  the **`client.c` `xcecli_*` layer**, not at `xce_request` and not at `xceapi.c`. Evidence: every
  consumer (worker `controller_connector.c`, scheduler `scheduler.c`, apiserver `clib.go`, xce_test)
  calls `xcecli_*`, never `xce_*` directly; and only `api_ctx_t` (client.c) holds the master list —
  `xce_ctx_t` (xceapi) does not. So `client.c` is both the universal surface and the only layer with
  conf+ep+ctx co-located. **`xce_*` stay one-shot/fail-fast and unchanged**; each `xcecli_*` wraps
  its `xce_*` call in a `do/while` driven by a shared `xcecli_retry_next` helper
  (`src/lib/client_failover.c` + `client_internal.h`). Trade-off accepted (user): ~33 wrappers gain
  a small loop (more LOC) but the policy is one helper and no async/thunk/plumbing changes — simpler
  and lower-risk. The orphaned `xce_request_with_retry()` is removed, not wired.
- **D2:** retire the no-op production branch of `xce_request_with_retry()`. Keep `xce_retry_config_t`
  + `XCE_RETRY_DEFAULTS`. Realign the unit tests (and hook contract) to the real status taxonomy
  and the round-trip shape (allowed by Req 8 / requirements decision; tracked in bug-608).
- **D3:** `handle_reply()` captures `reply.redirect_host/port` + status into the ctx (deep-copy the
  host string before the decoded reply is freed) so the failover loop can prefer the address.
- **D4:** on endpoint error/RDHUP, the err path must release the pending `cbuf_get` with a
  transport error so the loop reconnects instead of hanging.
- **D5:** controller populates `reply.redirect_host/port` from `ha.primary_host/port` at every
  `E_XCE_REDIRECT` site (join.c confirmed sole site; audit during implementation).
- **D6:** budget-exhaustion mapping — no-primary cause → return distinct `-E_XCE_NOT_READY`
  ("cluster not ready"); transport-only cause → return last transport error. Never `-EINVAL`.

### Decisions added after design review (validate-design pass)
- **D7 (CI1 — reply correlation):** `ctx->wbuf` is single-slot, status-only
  (`cbuf_init(...,1,sizeof(int))`, xcectx.c:185). `handle_reply` records `reply.seq` into the ctx;
  `xcecli_retry_next` drains any stale `wbuf` entry after reconnect (and reconnect closes the old fd
  first → no further old replies), so the next `xce_*` attempt cannot consume a prior-attempt reply.
- **D8 (CI2 — context state reset):** the existing transport-error path sets `ctx->state =
  XCE_CTX_ERR` (xcectx.c:148), which the per-op guard rejects as `-EINVAL` (xceapi.c:14,…) — the
  reported bug. `xcecli_retry_next` restores `XCE_CTX_RUN` after a successful reconnect, before the
  next `xce_*` attempt, so the unchanged per-op guard passes. `xceapi.c` is not modified.
### Simplification pass (goal: keep libxce easy to use)
- **D13 (redirect = hint, not contract):** the controller still populates the redirect address
  (forward-compatible, kept), but the client treats it as a **best-effort hint** — try it first if
  present, else rotate the master list; correctness never depends on it. This removes the heavy
  machinery the address-as-contract demanded: **drop the `reply_seq` correlation** (CI1 becomes
  drain-after-close only — reconnect closes the old fd first, then drain `wbuf`), and **collapse the
  classifier from 5 actions to 4** (`SUCCESS` / `RETRY_NOW` / `RETRY_BACKOFF` / `TERMINAL`) — the
  hint is consumed by `EndpointRotator`, not a separate classifier branch. Net: fewer fields, no
  lifetime-hazard dependency, simpler classification, same user-visible behavior. The rejoin hook
  and per-ctx lock are kept (real needs). Rationale: with 3 masters and rare failover, rotation is
  cheap; the hint is a latency nicety, not a correctness mechanism.

### Decisions added after 2nd design review (validate-design pass 2)
- **D10 (CI1 — op categories):** the ~33 `xcecli_*` ops split into reply-read (full failover,
  `is_write=false`), reply-write (full failover, `is_write=true`, DUP-replay→success), and
  fire-and-forget (`notify_*`/`heartbeat`/`wakeup` — `xce_*_req` only, no `cbuf_get`/status). The
  failover classifier applies only to reply-bearing ops; fire-and-forget gets transport-reconnect
  best-effort, and redelivery of lost notifies is deferred to `worker-rejoin-resync`.
- **D11 (CI2 — concurrency):** apiserver shares one ctx across goroutines (`clib.go:182`); add a
  `failover.lock` taken per `xcecli_*` op so reconnect/rotate cannot race another in-flight call.
- **D12 (rejoin hook — user requirement):** stateful daemons (apiserver/worker/scheduler) need a
  rejoin/state-exchange on reconnect; a worker also re-syncs its running-jobs state. libxce adds
  `xcecli_set_reconnect_handler()`; the failover loop invokes it after reconnect, before resume. The
  hook *content* is owned by `component-failover-adoption` (join) and `worker-rejoin-resync` (job
  state). Pure clients (`xce_test`) register none.

- **D9 (CI3 — idempotent-write replay):** job submit carries a **client-generated** `jobid`
  (`job.c:56`) with a server `-EEXIST`/`E_XCE_DUP_JOB` replay guard (`job.c:95`). The classifier maps
  `DUP_JOB` on a re-sent write (`is_write && attempt>0`) to success; on attempt 0 it is a genuine
  duplicate (terminal). This is what makes 5.2 actually satisfiable rather than assumed.

## Risks

- **R-async-blocking:** the round trip blocks longer under retry. Acceptable for CLI/apiserver
  (already blocking); for async C daemons it's the `component-failover-adoption` spec's concern
  (out of boundary here).
- **R-reply-routing on reconnect:** re-sending after reconnect must correlate the reply to the
  same ctx wait channel; mitigated by reusing the same `endpoint_t` (F5) and matching on `seq`.
- **R-redirect-lifetime:** `redirect_host` is a decoded capnp string freed with the reply; must be
  copied into a ctx-owned buffer (D3) to avoid use-after-free (cf. the `init_reply_by_req`
  memset hazard already documented at request.c:243-250).