127 lines
9.2 KiB
Markdown
127 lines
9.2 KiB
Markdown
# Research Log — libxce-failover
|
|
|
|
## Discovery Scope
|
|
|
|
Light, integration-focused discovery (extension of existing libxce). Goal: find the exact
|
|
seams to wire transparent failover into the client request/reply path. Verified by reading the
|
|
endpoint, client-config, request-core, reply, and controller-redirect code paths.
|
|
|
|
## Key Findings
|
|
|
|
### F1 — The async request/reply model determines the chokepoint (DECISIVE)
|
|
`xce_request(ep, req)` (request.c:317) only **encodes + enqueues** a packet — fire-and-forget.
|
|
The controller's reply (and its `status`) is delivered asynchronously: the endpoint's
|
|
`msg_handler` runs `handle_reply()` (xcectx.c:30) which signals the caller's wait channel, and
|
|
public ops block on `cbuf_get(&ctx->wbuf, &status)` (xceapi.c:116). Therefore the controller
|
|
status (`E_XCE_REDIRECT`/`NOMASTER`/`NOT_READY`/`STANDBY`) is observable only at the
|
|
**send-and-wait round trip**, not at `xce_request`. Failover orchestration must wrap that round
|
|
trip, where the `api_ctx_t` is in scope.
|
|
|
|
### F2 — `xce_request_with_retry()` production branch is a no-op stub
|
|
request.c:1972-1981: with no test hook installed, it does `xce_request(ep,req); return 0;`. The
|
|
entire retry/classify/backoff loop executes only under `xce_retry_send_hook` (tests). So the
|
|
"done" primitive cannot fail over against a real controller. Its hook contract
|
|
(`int hook(ep, req, reply_out)`) fakes a synchronous reply that production never produces.
|
|
|
|
### F3 — Redirect address is plumbed in the wire/struct but never populated or surfaced
|
|
- Wire schema has the fields: `controller.capnp:568-569` (`redirectHost`/`redirectPort`).
|
|
- Reply struct has them: `typemap.h:225-226` (`char *redirect_host; uint16_t redirect_port;`).
|
|
- Controller never sets them: every redirect site (`join.c:264`) sends `E_XCE_REDIRECT` with
|
|
`data_which = nodata`; grep finds **zero** production assignments to `reply.redirect_*`.
|
|
- Client never surfaces them: `handle_reply()` (xcectx.c:34) returns only `reply.status`; the
|
|
redirect fields are dropped before any caller sees them.
|
|
|
|
### F4 — Context already holds everything failover needs
|
|
`api_ctx_t` (client.c:23): `{ xce_client_config_t *conf; endpoint_t ep; xce_ctx_t ctx; }`.
|
|
`xce_client_config_t` (client.h:16): `masters[N_MAX_MASTERS]` + `master_ports[]` + `nmasters`.
|
|
So the master list, the endpoint to reconnect, and the reply channel are all reachable from one
|
|
context object. Missing: a current-master cursor, the retry config, and captured redirect target.
|
|
|
|
### F5 — Endpoint reconnect is straightforward and reusable
|
|
`endpoint_connect(ep, host, port)` (netio.c:659) does socket connect + SSL handshake on the
|
|
**same** `endpoint_t`. Reconnecting to a different master reuses the same endpoint object, so the
|
|
`msg_handler`/`owner` wiring that routes replies to `ctx->wbuf` stays intact.
|
|
|
|
### F6 — `xcecli_start()` only ever connects `masters[0]` (client.c:122)
|
|
No primary discovery at startup; single endpoint, never re-selected. Confirms the gap.
|
|
|
|
### F7 — Original EINVAL source
|
|
`xce_get_with_condition()` returns `-EINVAL` when `ctx->state != XCE_CTX_RUN` (xceapi.c:104).
|
|
After the primary dies the connection breaks and the ctx is left non-RUN, so every subsequent
|
|
call short-circuits to `-EINVAL` — the opaque error in the bug report. Failover must repair the
|
|
endpoint/ctx state instead of leaving it dead.
|
|
|
|
## Decisions (synthesis)
|
|
|
|
- **D1 (final — client-layer, supersedes the earlier xceapi round-trip idea):** failover lives in
|
|
the **`client.c` `xcecli_*` layer**, not at `xce_request` and not at `xceapi.c`. Evidence: every
|
|
consumer (worker `controller_connector.c`, scheduler `scheduler.c`, apiserver `clib.go`, xce_test)
|
|
calls `xcecli_*`, never `xce_*` directly; and only `api_ctx_t` (client.c) holds the master list —
|
|
`xce_ctx_t` (xceapi) does not. So `client.c` is both the universal surface and the only layer with
|
|
conf+ep+ctx co-located. **`xce_*` stay one-shot/fail-fast and unchanged**; each `xcecli_*` wraps
|
|
its `xce_*` call in a `do/while` driven by a shared `xcecli_retry_next` helper
|
|
(`src/lib/client_failover.c` + `client_internal.h`). Trade-off accepted (user): ~33 wrappers gain
|
|
a small loop (more LOC) but the policy is one helper and no async/thunk/plumbing changes — simpler
|
|
and lower-risk. The orphaned `xce_request_with_retry()` is removed, not wired.
|
|
- **D2:** retire the no-op production branch of `xce_request_with_retry()`. Keep `xce_retry_config_t`
|
|
+ `XCE_RETRY_DEFAULTS`. Realign the unit tests (and hook contract) to the real status taxonomy
|
|
and the round-trip shape (allowed by Req 8 / requirements decision; tracked in bug-608).
|
|
- **D3:** `handle_reply()` captures `reply.redirect_host/port` + status into the ctx (deep-copy the
|
|
host string before the decoded reply is freed) so the failover loop can prefer the address.
|
|
- **D4:** on endpoint error/RDHUP, the err path must release the pending `cbuf_get` with a
|
|
transport error so the loop reconnects instead of hanging.
|
|
- **D5:** controller populates `reply.redirect_host/port` from `ha.primary_host/port` at every
|
|
`E_XCE_REDIRECT` site (join.c confirmed sole site; audit during implementation).
|
|
- **D6:** budget-exhaustion mapping — no-primary cause → return distinct `-E_XCE_NOT_READY`
|
|
("cluster not ready"); transport-only cause → return last transport error. Never `-EINVAL`.
|
|
|
|
### Decisions added after design review (validate-design pass)
|
|
- **D7 (CI1 — reply correlation):** `ctx->wbuf` is single-slot, status-only
|
|
(`cbuf_init(...,1,sizeof(int))`, xcectx.c:185). `handle_reply` records `reply.seq` into the ctx;
|
|
`xcecli_retry_next` drains any stale `wbuf` entry after reconnect (and reconnect closes the old fd
|
|
first → no further old replies), so the next `xce_*` attempt cannot consume a prior-attempt reply.
|
|
- **D8 (CI2 — context state reset):** the existing transport-error path sets `ctx->state =
|
|
XCE_CTX_ERR` (xcectx.c:148), which the per-op guard rejects as `-EINVAL` (xceapi.c:14,…) — the
|
|
reported bug. `xcecli_retry_next` restores `XCE_CTX_RUN` after a successful reconnect, before the
|
|
next `xce_*` attempt, so the unchanged per-op guard passes. `xceapi.c` is not modified.
|
|
### Simplification pass (goal: keep libxce easy to use)
|
|
- **D13 (redirect = hint, not contract):** the controller still populates the redirect address
|
|
(forward-compatible, kept), but the client treats it as a **best-effort hint** — try it first if
|
|
present, else rotate the master list; correctness never depends on it. This removes the heavy
|
|
machinery the address-as-contract demanded: **drop the `reply_seq` correlation** (CI1 becomes
|
|
drain-after-close only — reconnect closes the old fd first, then drain `wbuf`), and **collapse the
|
|
classifier from 5 actions to 4** (`SUCCESS` / `RETRY_NOW` / `RETRY_BACKOFF` / `TERMINAL`) — the
|
|
hint is consumed by `EndpointRotator`, not a separate classifier branch. Net: fewer fields, no
|
|
lifetime-hazard dependency, simpler classification, same user-visible behavior. The rejoin hook
|
|
and per-ctx lock are kept (real needs). Rationale: with 3 masters and rare failover, rotation is
|
|
cheap; the hint is a latency nicety, not a correctness mechanism.
|
|
|
|
### Decisions added after 2nd design review (validate-design pass 2)
|
|
- **D10 (CI1 — op categories):** the ~33 `xcecli_*` ops split into reply-read (full failover,
|
|
`is_write=false`), reply-write (full failover, `is_write=true`, DUP-replay→success), and
|
|
fire-and-forget (`notify_*`/`heartbeat`/`wakeup` — `xce_*_req` only, no `cbuf_get`/status). The
|
|
failover classifier applies only to reply-bearing ops; fire-and-forget gets transport-reconnect
|
|
best-effort, and redelivery of lost notifies is deferred to `worker-rejoin-resync`.
|
|
- **D11 (CI2 — concurrency):** apiserver shares one ctx across goroutines (`clib.go:182`); add a
|
|
`failover.lock` taken per `xcecli_*` op so reconnect/rotate cannot race another in-flight call.
|
|
- **D12 (rejoin hook — user requirement):** stateful daemons (apiserver/worker/scheduler) need a
|
|
rejoin/state-exchange on reconnect; a worker also re-syncs its running-jobs state. libxce adds
|
|
`xcecli_set_reconnect_handler()`; the failover loop invokes it after reconnect, before resume. The
|
|
hook *content* is owned by `component-failover-adoption` (join) and `worker-rejoin-resync` (job
|
|
state). Pure clients (`xce_test`) register none.
|
|
|
|
- **D9 (CI3 — idempotent-write replay):** job submit carries a **client-generated** `jobid`
|
|
(`job.c:56`) with a server `-EEXIST`/`E_XCE_DUP_JOB` replay guard (`job.c:95`). The classifier maps
|
|
`DUP_JOB` on a re-sent write (`is_write && attempt>0`) to success; on attempt 0 it is a genuine
|
|
duplicate (terminal). This is what makes 5.2 actually satisfiable rather than assumed.
|
|
|
|
## Risks
|
|
|
|
- **R-async-blocking:** the round trip blocks longer under retry. Acceptable for CLI/apiserver
|
|
(already blocking); for async C daemons it's the `component-failover-adoption` spec's concern
|
|
(out of boundary here).
|
|
- **R-reply-routing on reconnect:** re-sending after reconnect must correlate the reply to the
|
|
same ctx wait channel; mitigated by reusing the same `endpoint_t` (F5) and matching on `seq`.
|
|
- **R-redirect-lifetime:** `redirect_host` is a decoded capnp string freed with the reply; must be
|
|
copied into a ctx-owned buffer (D3) to avoid use-after-free (cf. the `init_reply_by_req`
|
|
memset hazard already documented at request.c:243-250).
|