myxce/.kiro/specs/libxce-failover/research.md

# Research Log — libxce-failover

## Discovery Scope

Light, integration-focused discovery (extension of existing libxce). Goal: find the exact
seams to wire transparent failover into the client request/reply path. Verified by reading the
endpoint, client-config, request-core, reply, and controller-redirect code paths.

## Key Findings

### F1 — The async request/reply model determines the chokepoint (DECISIVE)
`xce_request(ep, req)` (request.c:317) only **encodes + enqueues** a packet — fire-and-forget.
The controller's reply (and its `status`) is delivered asynchronously: the endpoint's
`msg_handler` runs `handle_reply()` (xcectx.c:30) which signals the caller's wait channel, and
public ops block on `cbuf_get(&ctx->wbuf, &status)` (xceapi.c:116). Therefore the controller
status (`E_XCE_REDIRECT`/`NOMASTER`/`NOT_READY`/`STANDBY`) is observable only at the
**send-and-wait round trip**, not at `xce_request`. Failover orchestration must wrap that round
trip, where the `api_ctx_t` is in scope.

### F2 — `xce_request_with_retry()` production branch is a no-op stub
request.c:1972-1981: with no test hook installed, it does `xce_request(ep,req); return 0;`. The
entire retry/classify/backoff loop executes only under `xce_retry_send_hook` (tests). So the
"done" primitive cannot fail over against a real controller. Its hook contract
(`int hook(ep, req, reply_out)`) fakes a synchronous reply that production never produces.

### F3 — Redirect address is plumbed in the wire/struct but never populated or surfaced
- Wire schema has the fields: `controller.capnp:568-569` (`redirectHost`/`redirectPort`).
- Reply struct has them: `typemap.h:225-226` (`char *redirect_host; uint16_t redirect_port;`).
- Controller never sets them: every redirect site (`join.c:264`) sends `E_XCE_REDIRECT` with
  `data_which = nodata`; grep finds **zero** production assignments to `reply.redirect_*`.
- Client never surfaces them: `handle_reply()` (xcectx.c:34) returns only `reply.status`; the
  redirect fields are dropped before any caller sees them.

### F4 — Context already holds everything failover needs
`api_ctx_t` (client.c:23): `{ xce_client_config_t *conf; endpoint_t ep; xce_ctx_t ctx; }`.
`xce_client_config_t` (client.h:16): `masters[N_MAX_MASTERS]` + `master_ports[]` + `nmasters`.
So the master list, the endpoint to reconnect, and the reply channel are all reachable from one
context object. Missing: a current-master cursor, the retry config, and captured redirect target.

### F5 — Endpoint reconnect is straightforward and reusable
`endpoint_connect(ep, host, port)` (netio.c:659) does socket connect + SSL handshake on the
**same** `endpoint_t`. Reconnecting to a different master reuses the same endpoint object, so the
`msg_handler`/`owner` wiring that routes replies to `ctx->wbuf` stays intact.

### F6 — `xcecli_start()` only ever connects `masters[0]` (client.c:122)
No primary discovery at startup; single endpoint, never re-selected. Confirms the gap.

### F7 — Original EINVAL source
`xce_get_with_condition()` returns `-EINVAL` when `ctx->state != XCE_CTX_RUN` (xceapi.c:104).
After the primary dies the connection breaks and the ctx is left non-RUN, so every subsequent
call short-circuits to `-EINVAL` — the opaque error in the bug report. Failover must repair the
endpoint/ctx state instead of leaving it dead.

## Decisions (synthesis)

- **D1 (final — client-layer, supersedes the earlier xceapi round-trip idea):** failover lives in
  the **`client.c` `xcecli_*` layer**, not at `xce_request` and not at `xceapi.c`. Evidence: every
  consumer (worker `controller_connector.c`, scheduler `scheduler.c`, apiserver `clib.go`, xce_test)
  calls `xcecli_*`, never `xce_*` directly; and only `api_ctx_t` (client.c) holds the master list —
  `xce_ctx_t` (xceapi) does not. So `client.c` is both the universal surface and the only layer with
  conf+ep+ctx co-located. **`xce_*` stay one-shot/fail-fast and unchanged**; each `xcecli_*` wraps
  its `xce_*` call in a `do/while` driven by a shared `xcecli_retry_next` helper
  (`src/lib/client_failover.c` + `client_internal.h`). Trade-off accepted (user): ~33 wrappers gain
  a small loop (more LOC) but the policy is one helper and no async/thunk/plumbing changes — simpler
  and lower-risk. The orphaned `xce_request_with_retry()` is removed, not wired.
- **D2:** retire the no-op production branch of `xce_request_with_retry()`. Keep `xce_retry_config_t`
  + `XCE_RETRY_DEFAULTS`. Realign the unit tests (and hook contract) to the real status taxonomy
  and the round-trip shape (allowed by Req 8 / requirements decision; tracked in bug-608).
- **D3:** `handle_reply()` captures `reply.redirect_host/port` + status into the ctx (deep-copy the
  host string before the decoded reply is freed) so the failover loop can prefer the address.
- **D4:** on endpoint error/RDHUP, the err path must release the pending `cbuf_get` with a
  transport error so the loop reconnects instead of hanging.
- **D5:** controller populates `reply.redirect_host/port` from `ha.primary_host/port` at every
  `E_XCE_REDIRECT` site (join.c confirmed sole site; audit during implementation).
- **D6:** budget-exhaustion mapping — no-primary cause → return distinct `-E_XCE_NOT_READY`
  ("cluster not ready"); transport-only cause → return last transport error. Never `-EINVAL`.

### Decisions added after design review (validate-design pass)
- **D7 (CI1 — reply correlation):** `ctx->wbuf` is single-slot, status-only
  (`cbuf_init(...,1,sizeof(int))`, xcectx.c:185). `handle_reply` records `reply.seq` into the ctx;
  `xcecli_retry_next` drains any stale `wbuf` entry after reconnect (and reconnect closes the old fd
  first → no further old replies), so the next `xce_*` attempt cannot consume a prior-attempt reply.
- **D8 (CI2 — context state reset):** the existing transport-error path sets `ctx->state =
  XCE_CTX_ERR` (xcectx.c:148), which the per-op guard rejects as `-EINVAL` (xceapi.c:14,…) — the
  reported bug. `xcecli_retry_next` restores `XCE_CTX_RUN` after a successful reconnect, before the
  next `xce_*` attempt, so the unchanged per-op guard passes. `xceapi.c` is not modified.
### Simplification pass (goal: keep libxce easy to use)
- **D13 (redirect = hint, not contract):** the controller still populates the redirect address
  (forward-compatible, kept), but the client treats it as a **best-effort hint** — try it first if
  present, else rotate the master list; correctness never depends on it. This removes the heavy
  machinery the address-as-contract demanded: **drop the `reply_seq` correlation** (CI1 becomes
  drain-after-close only — reconnect closes the old fd first, then drain `wbuf`), and **collapse the
  classifier from 5 actions to 4** (`SUCCESS` / `RETRY_NOW` / `RETRY_BACKOFF` / `TERMINAL`) — the
  hint is consumed by `EndpointRotator`, not a separate classifier branch. Net: fewer fields, no
  lifetime-hazard dependency, simpler classification, same user-visible behavior. The rejoin hook
  and per-ctx lock are kept (real needs). Rationale: with 3 masters and rare failover, rotation is
  cheap; the hint is a latency nicety, not a correctness mechanism.

### Decisions added after 2nd design review (validate-design pass 2)
- **D10 (CI1 — op categories):** the ~33 `xcecli_*` ops split into reply-read (full failover,
  `is_write=false`), reply-write (full failover, `is_write=true`, DUP-replay→success), and
  fire-and-forget (`notify_*`/`heartbeat`/`wakeup` — `xce_*_req` only, no `cbuf_get`/status). The
  failover classifier applies only to reply-bearing ops; fire-and-forget gets transport-reconnect
  best-effort, and redelivery of lost notifies is deferred to `worker-rejoin-resync`.
- **D11 (CI2 — concurrency):** apiserver shares one ctx across goroutines (`clib.go:182`); add a
  `failover.lock` taken per `xcecli_*` op so reconnect/rotate cannot race another in-flight call.
- **D12 (rejoin hook — user requirement):** stateful daemons (apiserver/worker/scheduler) need a
  rejoin/state-exchange on reconnect; a worker also re-syncs its running-jobs state. libxce adds
  `xcecli_set_reconnect_handler()`; the failover loop invokes it after reconnect, before resume. The
  hook *content* is owned by `component-failover-adoption` (join) and `worker-rejoin-resync` (job
  state). Pure clients (`xce_test`) register none.

- **D9 (CI3 — idempotent-write replay):** job submit carries a **client-generated** `jobid`
  (`job.c:56`) with a server `-EEXIST`/`E_XCE_DUP_JOB` replay guard (`job.c:95`). The classifier maps
  `DUP_JOB` on a re-sent write (`is_write && attempt>0`) to success; on attempt 0 it is a genuine
  duplicate (terminal). This is what makes 5.2 actually satisfiable rather than assumed.

## Risks

- **R-async-blocking:** the round trip blocks longer under retry. Acceptable for CLI/apiserver
  (already blocking); for async C daemons it's the `component-failover-adoption` spec's concern
  (out of boundary here).
- **R-reply-routing on reconnect:** re-sending after reconnect must correlate the reply to the
  same ctx wait channel; mitigated by reusing the same `endpoint_t` (F5) and matching on `seq`.
- **R-redirect-lifetime:** `redirect_host` is a decoded capnp string freed with the reply; must be
  copied into a ctx-owned buffer (D3) to avoid use-after-free (cf. the `init_reply_by_req`
  memset hazard already documented at request.c:243-250).

---

## Gap Re-Audit (2026-06-18)

**Method:** Forward-looking requirements-vs-codebase verification. Each requirement traced to a
real symbol/file:line in `src/lib`, `src/include`, `src/controller`. Every `[x]` task confirmed
against actual code (not just the checkbox). The only genuinely-open leaf task (6.2) re-scoped for
the remaining design/exec work.

### Verdict

**STATUS: PARTIAL — code-complete, live-E2E-incomplete.** All implementation tasks (1.1–6.1 + 5.3)
are present and wired in real code; the sole open leaf is **6.2 (live 3-node failover E2E)**, which
Implementation Notes flag as *partial* (server-side failover proven + 2 live bugs found/fixed, but
clean within-session client transparent-recovery never visually demonstrated in the sandbox env).
The 6 unchecked `[ ]` items besides 6.2 are **phase-group headers** (tasks 1/2/3/4/5/6), not work
units — all their leaf sub-tasks are `[x]` and verified below.

### Requirement-to-Asset Map

| Req | Summary | Tag | Evidence (file:line) |
|-----|---------|-----|----------------------|
| 1.1 | All ops (read+write) through failover | **MET** | `client.c` — 21× `XCECLI_FAILOVER_CALL`/`XCECLI_FF_CALL`; macro in `client_internal.h:152` |
| 1.2 | Next request completes on new primary | **MET** (unit) / **Unknown** (live) | `client_failover.c:xcecli_retry_next` + `xce_ctx_reconnect` (`xcectx.c:320`); live not visually confirmed (6.2) |
| 1.3 | Public API signatures unchanged | **MET** | Notes 5.1 "zero signature changes"; macro wraps op bodies, no proto edits |
| 2.1 | Unreachable → try other masters | **MET** | `xce_failover_next_target` (cursor advance+wrap over `conf->masters[]`), `client_failover.c` |
| 2.2 | Redirect → rotate, hint-first | **MET** | hint-first/consume then cursor; `xce_failover_rotate` |
| 2.3 | Standby → treat as redirect/rotate | **MET** | classifier `client_failover.c:59` (`E_XCE_STANDBY`→retry-now) |
| 2.4 | Sticky current primary | **MET** | `cur_master` in `xce_failover_state_t` (`client_internal.h`) |
| 2.5 | Redirect/standby/no-primary ≠ success | **MET** | classifier never maps these to SUCCESS; unit `test_libxce_retry.c` |
| 3.1 | No-primary → backoff to budget | **MET** | `client_failover.c:64` (NOMASTER/NOT_READY→backoff) |
| 3.2 | Success before budget | **MET** (unit) | `test_libxce_retry.c` become-primary-mid-loop |
| 3.3/3.4 | Budget exhausted → distinct cluster-not-ready (not EINVAL) | **MET** | `*rc = -E_XCE_NOT_READY` (`client_failover.c:282,317`) |
| 4.1–4.5 | Retriable vs terminal classification | **MET** | `xce_failover_classify`, 11-row taxonomy in `failover_harness.h`; tests pass |
| 5.1 | Reads+writes failover | **MET** | OpCategory: writes=true for jobsub/jobctrl/join/… (Notes 5.1) |
| 5.2 | Write retry not double-applied (idempotent DUP→success) | **MET** (client) / **Constraint** | classifier DUP-on-resent-write→success; relies on controller `job.c:56,95` jobid replay guard |
| 5.3 | No silent drop | **MET** | loop returns success or distinct error |
| 6.1/6.2 | Configurable params + safe defaults | **MET** | `xce_retry_config_t` + `XCE_RETRY_DEFAULTS` (`request.h:183-193`, max_retries=20) |
| 6.3 | Jitter / anti-thundering-herd | **MET** | `test_decide_jitter_staggers_many_clients` (64-carrier, Notes 6.1) |
| 7.1/7.2 | Log rotation target + exhaustion count | **MET** | `tlog_info`/`tlog_warn` in `xcecli_retry_next` (Notes 3.1) |
| 8.1 | Redirect carries primary host/port | **MET** | `fill_redirect_reply` (`join.c:28-31`) ← `ha.primary_host/port` (`ha.h:149-150`) |
| 8.2 | Unknown primary → no-primary, not addressless redirect | **MET** | `join.c:289` `-E_XCE_NOMASTER` path unchanged |
| 8.3/8.4 | Client uses hint best-effort, falls back to rotation | **MET** | `xce_ctx_capture_redirect_hint` (`xcectx.c:30,57`); rotate consumes-then-falls-through |

**Orphan cleanup (task 1.1):** confirmed removed — `xce_request_with_retry()` gone, only a
tombstone comment remains (`request.c:1927`).
**Controller audit (task 5.3):** `join.c` confirmed sole `E_XCE_REDIRECT` emitter; controller test
`test_join_redirect.c` registered (`controller_test.c:155`, `meson.build:97`).

Tally: **24/24 acceptance criteria MET in code** (1.2 live-unconfirmed; 5.2 carries one external
Constraint = controller idempotency, which is in-scope-consumed not in-scope-built).

### Open Tasks — Forward Gap Analysis

Only one leaf task is open. The remaining `[ ]` are phase headers that auto-close when their
sub-tasks are `[x]` (all are) — recommend marking 1/2/3/4/5 complete; keep 6 open until 6.2 lands.

#### Task 6.2 — Live 3-node failover end-to-end  *(the real PENDING work)*

**What is missing (vs. what is done):**
- DONE in-env: 3-node `deploy-ha.sh` cluster elected a primary; killing it triggered server-side
  failover (ctrl-2 → term=2); the run REPRODUCED the original `Invalid argument` bug and the failover
  loop now ENGAGES (`xcecli_joblist`→`xcecli_retry_next`→`xce_failover_rotate`). Two live-only bugs
  found+fixed with regression unit tests: **bug-609** (fresh-call-after-disconnect `-EINVAL`
  classified terminal → known-dead-ctx override in `decide`) and **bug-610** (`xce_failover_rotate`
  SEGV — reconnect never re-bound `ep->iomgr`/handlers; fixed via new `xce_ctx_reconnect()`).
- MISSING: a clean, *visually-captured* within-session demonstration of the three acceptance
  outcomes against a stable runtime: (a) `joblist` reports cluster-not-ready during the election
  window then transparently recovers with NO `EINVAL`; (b) a request to a standby lands on the
  primary in one hop via the populated redirect hint (and via rotation when hint absent); (c) a
  `jobsub` spanning the failover creates **exactly one** job (idempotency end-to-end).

**Integration points to exercise:** `/xce-deploy-ha localhost:9991 9992 9993` → `xce_test`
(`set masters`, `client`, `jobsub`, `joblist`) → kill primary → re-`joblist`. NOTE the build hazard
recorded twice in Notes: **building `lib` removes the controller binary from the shared build dir** —
must `xce-build.sh controller` (relinks `xce_test`) before each deploy.

**Options:**
- **Option A (recommended): stable-env manual E2E.** Run the reproduction in a non-sandbox env where
  backgrounded controller procs survive between steps (the sandbox reaps them). Capture the 3 outcomes.
  *Effort S–M, Risk Low* — code already proven to engage; this is confirmation + evidence capture.
- **Option B: scripted deterministic E2E harness** (a `testsuite` shell/expect script that boots the
  cluster, kills the primary on a signal, asserts on `xce_test` output). *Effort M, Risk Med* — gives
  a repeatable regression gate but adds harness maintenance; process-lifecycle flakiness is the risk.
- **Option C: in-process integration test** simulating the controller side with fakes (no real procs).
  *Effort M–L, Risk Med* — avoids deploy flakiness but re-tests what unit tests already cover (low
  marginal value vs. real-cluster confidence); does NOT satisfy 6.2's "original reproduction" intent.

**Recommendation:** Option A to *close* 6.2 (intent = the original bug reproduction passes live),
optionally backstopped by Option B as a CI regression gate in a follow-on.

#### Phase headers 1–5 (and 6) — administrative

*Gap:* checkbox-only. *Action:* flip 1/2/3/4/5 to `[x]` (all sub-tasks verified `[x]` above); leave
6 `[ ]` until 6.2 closes. *Effort S, Risk None.*

### Research-Needed (for any remaining design/exec)

1. **Stable runtime for live E2E** — identify an env where backgrounded `xce_controller` procs
   persist across orchestration steps (sandbox reaps them; nix runtime env needed). Blocks 6.2-A.
2. **Build-order guard** — codify "rebuild controller after any `lib` build" (Notes 3.1/6.2 twice).
   Consider a `xce-build.sh` ordering note or a make dependency so `xce_test` is never stale-linked.
3. **Idempotency end-to-end proof (5.2)** — confirm the controller's client-`jobid` replay guard
   (`job.c:56,95`) actually returns `E_XCE_DUP_JOB`/`-EEXIST` on the *re-sent* write across a real
   failover (unit-mocked today; live single-job-created assertion is the open evidence).
4. **Downstream readiness (out-of-boundary, flag only)** — `component-failover-adoption` (HARD dep)
   still owes: per-daemon **full master-list** wiring (today even CLI uses `masters[0]`,
   `client.c:131`) + **liveness-safe** bounded-blocking verification for async C daemons (worker
   `controller_connector.c`, scheduler) so backoff never stalls the heartbeat/iomgr thread. Not 6.2's
   scope, but it is the gate before this failover mechanism delivers value to real daemons.

### Integration Risks (remaining)

- **Process-lifecycle flakiness** in any live/scripted E2E (controller procs, port reuse, election
  timing) — the dominant risk for 6.2; mitigated by Option A in a stable env + jittered election.
- **Stale `xce_test` link** if controller isn't rebuilt after a `lib` build — would mask a real fix
  or surface a phantom one. Mitigation = Research-Needed #2.
- **Idempotency dependency** (5.2) is external (controller `job.c`) — a regression there breaks
  exactly-once across failover without touching libxce. Mitigation = Research-Needed #3.