lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/libxce-failover/research.md

9.2 KiB
Raw Blame History

Research Log — libxce-failover

Discovery Scope

Light, integration-focused discovery (extension of existing libxce). Goal: find the exact seams to wire transparent failover into the client request/reply path. Verified by reading the endpoint, client-config, request-core, reply, and controller-redirect code paths.

Key Findings

F1 — The async request/reply model determines the chokepoint (DECISIVE)

xce_request(ep, req) (request.c:317) only encodes + enqueues a packet — fire-and-forget. The controller's reply (and its status) is delivered asynchronously: the endpoint's msg_handler runs handle_reply() (xcectx.c:30) which signals the caller's wait channel, and public ops block on cbuf_get(&ctx->wbuf, &status) (xceapi.c:116). Therefore the controller status (E_XCE_REDIRECT/NOMASTER/NOT_READY/STANDBY) is observable only at the send-and-wait round trip, not at xce_request. Failover orchestration must wrap that round trip, where the api_ctx_t is in scope.

F2 — xce_request_with_retry() production branch is a no-op stub

request.c:1972-1981: with no test hook installed, it does xce_request(ep,req); return 0;. The entire retry/classify/backoff loop executes only under xce_retry_send_hook (tests). So the "done" primitive cannot fail over against a real controller. Its hook contract (int hook(ep, req, reply_out)) fakes a synchronous reply that production never produces.

F3 — Redirect address is plumbed in the wire/struct but never populated or surfaced

  • Wire schema has the fields: controller.capnp:568-569 (redirectHost/redirectPort).
  • Reply struct has them: typemap.h:225-226 (char *redirect_host; uint16_t redirect_port;).
  • Controller never sets them: every redirect site (join.c:264) sends E_XCE_REDIRECT with data_which = nodata; grep finds zero production assignments to reply.redirect_*.
  • Client never surfaces them: handle_reply() (xcectx.c:34) returns only reply.status; the redirect fields are dropped before any caller sees them.

F4 — Context already holds everything failover needs

api_ctx_t (client.c:23): { xce_client_config_t *conf; endpoint_t ep; xce_ctx_t ctx; }. xce_client_config_t (client.h:16): masters[N_MAX_MASTERS] + master_ports[] + nmasters. So the master list, the endpoint to reconnect, and the reply channel are all reachable from one context object. Missing: a current-master cursor, the retry config, and captured redirect target.

F5 — Endpoint reconnect is straightforward and reusable

endpoint_connect(ep, host, port) (netio.c:659) does socket connect + SSL handshake on the same endpoint_t. Reconnecting to a different master reuses the same endpoint object, so the msg_handler/owner wiring that routes replies to ctx->wbuf stays intact.

F6 — xcecli_start() only ever connects masters[0] (client.c:122)

No primary discovery at startup; single endpoint, never re-selected. Confirms the gap.

F7 — Original EINVAL source

xce_get_with_condition() returns -EINVAL when ctx->state != XCE_CTX_RUN (xceapi.c:104). After the primary dies the connection breaks and the ctx is left non-RUN, so every subsequent call short-circuits to -EINVAL — the opaque error in the bug report. Failover must repair the endpoint/ctx state instead of leaving it dead.

Decisions (synthesis)

  • D1 (final — client-layer, supersedes the earlier xceapi round-trip idea): failover lives in the client.c xcecli_* layer, not at xce_request and not at xceapi.c. Evidence: every consumer (worker controller_connector.c, scheduler scheduler.c, apiserver clib.go, xce_test) calls xcecli_*, never xce_* directly; and only api_ctx_t (client.c) holds the master list — xce_ctx_t (xceapi) does not. So client.c is both the universal surface and the only layer with conf+ep+ctx co-located. xce_* stay one-shot/fail-fast and unchanged; each xcecli_* wraps its xce_* call in a do/while driven by a shared xcecli_retry_next helper (src/lib/client_failover.c + client_internal.h). Trade-off accepted (user): ~33 wrappers gain a small loop (more LOC) but the policy is one helper and no async/thunk/plumbing changes — simpler and lower-risk. The orphaned xce_request_with_retry() is removed, not wired.
  • D2: retire the no-op production branch of xce_request_with_retry(). Keep xce_retry_config_t
    • XCE_RETRY_DEFAULTS. Realign the unit tests (and hook contract) to the real status taxonomy and the round-trip shape (allowed by Req 8 / requirements decision; tracked in bug-608).
  • D3: handle_reply() captures reply.redirect_host/port + status into the ctx (deep-copy the host string before the decoded reply is freed) so the failover loop can prefer the address.
  • D4: on endpoint error/RDHUP, the err path must release the pending cbuf_get with a transport error so the loop reconnects instead of hanging.
  • D5: controller populates reply.redirect_host/port from ha.primary_host/port at every E_XCE_REDIRECT site (join.c confirmed sole site; audit during implementation).
  • D6: budget-exhaustion mapping — no-primary cause → return distinct -E_XCE_NOT_READY ("cluster not ready"); transport-only cause → return last transport error. Never -EINVAL.

Decisions added after design review (validate-design pass)

  • D7 (CI1 — reply correlation): ctx->wbuf is single-slot, status-only (cbuf_init(...,1,sizeof(int)), xcectx.c:185). handle_reply records reply.seq into the ctx; xcecli_retry_next drains any stale wbuf entry after reconnect (and reconnect closes the old fd first → no further old replies), so the next xce_* attempt cannot consume a prior-attempt reply.
  • D8 (CI2 — context state reset): the existing transport-error path sets ctx->state = XCE_CTX_ERR (xcectx.c:148), which the per-op guard rejects as -EINVAL (xceapi.c:14,…) — the reported bug. xcecli_retry_next restores XCE_CTX_RUN after a successful reconnect, before the next xce_* attempt, so the unchanged per-op guard passes. xceapi.c is not modified.

Simplification pass (goal: keep libxce easy to use)

  • D13 (redirect = hint, not contract): the controller still populates the redirect address (forward-compatible, kept), but the client treats it as a best-effort hint — try it first if present, else rotate the master list; correctness never depends on it. This removes the heavy machinery the address-as-contract demanded: drop the reply_seq correlation (CI1 becomes drain-after-close only — reconnect closes the old fd first, then drain wbuf), and collapse the classifier from 5 actions to 4 (SUCCESS / RETRY_NOW / RETRY_BACKOFF / TERMINAL) — the hint is consumed by EndpointRotator, not a separate classifier branch. Net: fewer fields, no lifetime-hazard dependency, simpler classification, same user-visible behavior. The rejoin hook and per-ctx lock are kept (real needs). Rationale: with 3 masters and rare failover, rotation is cheap; the hint is a latency nicety, not a correctness mechanism.

Decisions added after 2nd design review (validate-design pass 2)

  • D10 (CI1 — op categories): the ~33 xcecli_* ops split into reply-read (full failover, is_write=false), reply-write (full failover, is_write=true, DUP-replay→success), and fire-and-forget (notify_*/heartbeat/wakeup — xce_*_req only, no cbuf_get/status). The failover classifier applies only to reply-bearing ops; fire-and-forget gets transport-reconnect best-effort, and redelivery of lost notifies is deferred to worker-rejoin-resync.

  • D11 (CI2 — concurrency): apiserver shares one ctx across goroutines (clib.go:182); add a failover.lock taken per xcecli_* op so reconnect/rotate cannot race another in-flight call.

  • D12 (rejoin hook — user requirement): stateful daemons (apiserver/worker/scheduler) need a rejoin/state-exchange on reconnect; a worker also re-syncs its running-jobs state. libxce adds xcecli_set_reconnect_handler(); the failover loop invokes it after reconnect, before resume. The hook content is owned by component-failover-adoption (join) and worker-rejoin-resync (job state). Pure clients (xce_test) register none.

  • D9 (CI3 — idempotent-write replay): job submit carries a client-generated jobid (job.c:56) with a server -EEXIST/E_XCE_DUP_JOB replay guard (job.c:95). The classifier maps DUP_JOB on a re-sent write (is_write && attempt>0) to success; on attempt 0 it is a genuine duplicate (terminal). This is what makes 5.2 actually satisfiable rather than assumed.

Risks

  • R-async-blocking: the round trip blocks longer under retry. Acceptable for CLI/apiserver (already blocking); for async C daemons it's the component-failover-adoption spec's concern (out of boundary here).
  • R-reply-routing on reconnect: re-sending after reconnect must correlate the reply to the same ctx wait channel; mitigated by reusing the same endpoint_t (F5) and matching on seq.
  • R-redirect-lifetime: redirect_host is a decoded capnp string freed with the reply; must be copied into a ctx-owned buffer (D3) to avoid use-after-free (cf. the init_reply_by_req memset hazard already documented at request.c:243-250).