myxce/.kiro/specs/libxce-failover/research.md

Research Log — libxce-failover

Discovery Scope

Light, integration-focused discovery (extension of existing libxce). Goal: find the exact seams to wire transparent failover into the client request/reply path. Verified by reading the endpoint, client-config, request-core, reply, and controller-redirect code paths.

Key Findings

F1 — The async request/reply model determines the chokepoint (DECISIVE)

xce_request(ep, req) (request.c:317) only encodes + enqueues a packet — fire-and-forget. The controller's reply (and its status) is delivered asynchronously: the endpoint's msg_handler runs handle_reply() (xcectx.c:30) which signals the caller's wait channel, and public ops block on cbuf_get(&ctx->wbuf, &status) (xceapi.c:116). Therefore the controller status (E_XCE_REDIRECT/NOMASTER/NOT_READY/STANDBY) is observable only at the send-and-wait round trip, not at xce_request. Failover orchestration must wrap that round trip, where the api_ctx_t is in scope.

F2 — xce_request_with_retry() production branch is a no-op stub

request.c:1972-1981: with no test hook installed, it does xce_request(ep,req); return 0;. The entire retry/classify/backoff loop executes only under xce_retry_send_hook (tests). So the "done" primitive cannot fail over against a real controller. Its hook contract (int hook(ep, req, reply_out)) fakes a synchronous reply that production never produces.

F3 — Redirect address is plumbed in the wire/struct but never populated or surfaced

• Wire schema has the fields: controller.capnp:568-569 (redirectHost/redirectPort).
• Reply struct has them: typemap.h:225-226 (char *redirect_host; uint16_t redirect_port;).
• Controller never sets them: every redirect site (join.c:264) sends E_XCE_REDIRECT with data_which = nodata; grep finds zero production assignments to reply.redirect_*.
• Client never surfaces them: handle_reply() (xcectx.c:34) returns only reply.status; the redirect fields are dropped before any caller sees them.

F4 — Context already holds everything failover needs

api_ctx_t (client.c:23): { xce_client_config_t *conf; endpoint_t ep; xce_ctx_t ctx; }. xce_client_config_t (client.h:16): masters[N_MAX_MASTERS] + master_ports[] + nmasters. So the master list, the endpoint to reconnect, and the reply channel are all reachable from one context object. Missing: a current-master cursor, the retry config, and captured redirect target.

F5 — Endpoint reconnect is straightforward and reusable

endpoint_connect(ep, host, port) (netio.c:659) does socket connect + SSL handshake on the same endpoint_t. Reconnecting to a different master reuses the same endpoint object, so the msg_handler/owner wiring that routes replies to ctx->wbuf stays intact.

F6 — xcecli_start() only ever connects masters[0] (client.c:122)

No primary discovery at startup; single endpoint, never re-selected. Confirms the gap.

F7 — Original EINVAL source

xce_get_with_condition() returns -EINVAL when ctx->state != XCE_CTX_RUN (xceapi.c:104). After the primary dies the connection breaks and the ctx is left non-RUN, so every subsequent call short-circuits to -EINVAL — the opaque error in the bug report. Failover must repair the endpoint/ctx state instead of leaving it dead.

Decisions (synthesis)

• D1 (final — client-layer, supersedes the earlier xceapi round-trip idea): failover lives in the client.c xcecli_* layer, not at xce_request and not at xceapi.c. Evidence: every consumer (worker controller_connector.c, scheduler scheduler.c, apiserver clib.go, xce_test) calls xcecli_*, never xce_* directly; and only api_ctx_t (client.c) holds the master list — xce_ctx_t (xceapi) does not. So client.c is both the universal surface and the only layer with conf+ep+ctx co-located. xce_* stay one-shot/fail-fast and unchanged; each xcecli_* wraps its xce_* call in a do/while driven by a shared xcecli_retry_next helper (src/lib/client_failover.c + client_internal.h). Trade-off accepted (user): ~33 wrappers gain a small loop (more LOC) but the policy is one helper and no async/thunk/plumbing changes — simpler and lower-risk. The orphaned xce_request_with_retry() is removed, not wired.
• D2: retire the no-op production branch of xce_request_with_retry(). Keep xce_retry_config_t
  • XCE_RETRY_DEFAULTS. Realign the unit tests (and hook contract) to the real status taxonomy and the round-trip shape (allowed by Req 8 / requirements decision; tracked in bug-608).
• D3: handle_reply() captures reply.redirect_host/port + status into the ctx (deep-copy the host string before the decoded reply is freed) so the failover loop can prefer the address.
• D4: on endpoint error/RDHUP, the err path must release the pending cbuf_get with a transport error so the loop reconnects instead of hanging.
• D5: controller populates reply.redirect_host/port from ha.primary_host/port at every E_XCE_REDIRECT site (join.c confirmed sole site; audit during implementation).
• D6: budget-exhaustion mapping — no-primary cause → return distinct -E_XCE_NOT_READY ("cluster not ready"); transport-only cause → return last transport error. Never -EINVAL.

Decisions added after design review (validate-design pass)

• D7 (CI1 — reply correlation): ctx->wbuf is single-slot, status-only (cbuf_init(...,1,sizeof(int)), xcectx.c:185). handle_reply records reply.seq into the ctx; xcecli_retry_next drains any stale wbuf entry after reconnect (and reconnect closes the old fd first → no further old replies), so the next xce_* attempt cannot consume a prior-attempt reply.
• D8 (CI2 — context state reset): the existing transport-error path sets ctx->state = XCE_CTX_ERR (xcectx.c:148), which the per-op guard rejects as -EINVAL (xceapi.c:14,…) — the reported bug. xcecli_retry_next restores XCE_CTX_RUN after a successful reconnect, before the next xce_* attempt, so the unchanged per-op guard passes. xceapi.c is not modified.

Simplification pass (goal: keep libxce easy to use)

• D13 (redirect = hint, not contract): the controller still populates the redirect address (forward-compatible, kept), but the client treats it as a best-effort hint — try it first if present, else rotate the master list; correctness never depends on it. This removes the heavy machinery the address-as-contract demanded: drop the reply_seq correlation (CI1 becomes drain-after-close only — reconnect closes the old fd first, then drain wbuf), and collapse the classifier from 5 actions to 4 (SUCCESS / RETRY_NOW / RETRY_BACKOFF / TERMINAL) — the hint is consumed by EndpointRotator, not a separate classifier branch. Net: fewer fields, no lifetime-hazard dependency, simpler classification, same user-visible behavior. The rejoin hook and per-ctx lock are kept (real needs). Rationale: with 3 masters and rare failover, rotation is cheap; the hint is a latency nicety, not a correctness mechanism.

Decisions added after 2nd design review (validate-design pass 2)

• D10 (CI1 — op categories): the ~33 xcecli_* ops split into reply-read (full failover, is_write=false), reply-write (full failover, is_write=true, DUP-replay→success), and fire-and-forget (notify_*/heartbeat/wakeup — xce_*_req only, no cbuf_get/status). The failover classifier applies only to reply-bearing ops; fire-and-forget gets transport-reconnect best-effort, and redelivery of lost notifies is deferred to worker-rejoin-resync.

• D11 (CI2 — concurrency): apiserver shares one ctx across goroutines (clib.go:182); add a failover.lock taken per xcecli_* op so reconnect/rotate cannot race another in-flight call.

• D12 (rejoin hook — user requirement): stateful daemons (apiserver/worker/scheduler) need a rejoin/state-exchange on reconnect; a worker also re-syncs its running-jobs state. libxce adds xcecli_set_reconnect_handler(); the failover loop invokes it after reconnect, before resume. The hook content is owned by component-failover-adoption (join) and worker-rejoin-resync (job state). Pure clients (xce_test) register none.

• D9 (CI3 — idempotent-write replay): job submit carries a client-generated jobid (job.c:56) with a server -EEXIST/E_XCE_DUP_JOB replay guard (job.c:95). The classifier maps DUP_JOB on a re-sent write (is_write && attempt>0) to success; on attempt 0 it is a genuine duplicate (terminal). This is what makes 5.2 actually satisfiable rather than assumed.

Risks

• R-async-blocking: the round trip blocks longer under retry. Acceptable for CLI/apiserver (already blocking); for async C daemons it's the component-failover-adoption spec's concern (out of boundary here).
• R-reply-routing on reconnect: re-sending after reconnect must correlate the reply to the same ctx wait channel; mitigated by reusing the same endpoint_t (F5) and matching on seq.
• R-redirect-lifetime: redirect_host is a decoded capnp string freed with the reply; must be copied into a ctx-owned buffer (D3) to avoid use-after-free (cf. the init_reply_by_req memset hazard already documented at request.c:243-250).

---

Gap Re-Audit (2026-06-18)

Method: Forward-looking requirements-vs-codebase verification. Each requirement traced to a real symbol/file:line in src/lib, src/include, src/controller. Every [x] task confirmed against actual code (not just the checkbox). The only genuinely-open leaf task (6.2) re-scoped for the remaining design/exec work.

Verdict

STATUS: PARTIAL — code-complete, live-E2E-incomplete. All implementation tasks (1.1–6.1 + 5.3) are present and wired in real code; the sole open leaf is 6.2 (live 3-node failover E2E), which Implementation Notes flag as partial (server-side failover proven + 2 live bugs found/fixed, but clean within-session client transparent-recovery never visually demonstrated in the sandbox env). The 6 unchecked [ ] items besides 6.2 are phase-group headers (tasks 1/2/3/4/5/6), not work units — all their leaf sub-tasks are [x] and verified below.

Requirement-to-Asset Map

Req	Summary	Tag	Evidence (file:line)
1.1	All ops (read+write) through failover	MET	client.c — 21× XCECLI_FAILOVER_CALL/XCECLI_FF_CALL; macro in client_internal.h:152
1.2	Next request completes on new primary	MET (unit) / Unknown (live)	client_failover.c:xcecli_retry_next + xce_ctx_reconnect (xcectx.c:320); live not visually confirmed (6.2)
1.3	Public API signatures unchanged	MET	Notes 5.1 "zero signature changes"; macro wraps op bodies, no proto edits
2.1	Unreachable → try other masters	MET	xce_failover_next_target (cursor advance+wrap over conf->masters[]), client_failover.c
2.2	Redirect → rotate, hint-first	MET	hint-first/consume then cursor; xce_failover_rotate
2.3	Standby → treat as redirect/rotate	MET	classifier client_failover.c:59 (E_XCE_STANDBY→retry-now)
2.4	Sticky current primary	MET	cur_master in xce_failover_state_t (client_internal.h)
2.5	Redirect/standby/no-primary ≠ success	MET	classifier never maps these to SUCCESS; unit test_libxce_retry.c
3.1	No-primary → backoff to budget	MET	client_failover.c:64 (NOMASTER/NOT_READY→backoff)
3.2	Success before budget	MET (unit)	test_libxce_retry.c become-primary-mid-loop
3.3/3.4	Budget exhausted → distinct cluster-not-ready (not EINVAL)	MET	*rc = -E_XCE_NOT_READY (client_failover.c:282,317)
4.1–4.5	Retriable vs terminal classification	MET	xce_failover_classify, 11-row taxonomy in failover_harness.h; tests pass
5.1	Reads+writes failover	MET	OpCategory: writes=true for jobsub/jobctrl/join/… (Notes 5.1)
5.2	Write retry not double-applied (idempotent DUP→success)	MET (client) / Constraint	classifier DUP-on-resent-write→success; relies on controller job.c:56,95 jobid replay guard
5.3	No silent drop	MET	loop returns success or distinct error
6.1/6.2	Configurable params + safe defaults	MET	xce_retry_config_t + XCE_RETRY_DEFAULTS (request.h:183-193, max_retries=20)
6.3	Jitter / anti-thundering-herd	MET	test_decide_jitter_staggers_many_clients (64-carrier, Notes 6.1)
7.1/7.2	Log rotation target + exhaustion count	MET	tlog_info/tlog_warn in xcecli_retry_next (Notes 3.1)
8.1	Redirect carries primary host/port	MET	fill_redirect_reply (join.c:28-31) ← ha.primary_host/port (ha.h:149-150)
8.2	Unknown primary → no-primary, not addressless redirect	MET	join.c:289 -E_XCE_NOMASTER path unchanged
8.3/8.4	Client uses hint best-effort, falls back to rotation	MET	xce_ctx_capture_redirect_hint (xcectx.c:30,57); rotate consumes-then-falls-through

Orphan cleanup (task 1.1): confirmed removed — xce_request_with_retry() gone, only a tombstone comment remains (request.c:1927). Controller audit (task 5.3): join.c confirmed sole E_XCE_REDIRECT emitter; controller test test_join_redirect.c registered (controller_test.c:155, meson.build:97).

Tally: 24/24 acceptance criteria MET in code (1.2 live-unconfirmed; 5.2 carries one external Constraint = controller idempotency, which is in-scope-consumed not in-scope-built).

Open Tasks — Forward Gap Analysis

Only one leaf task is open. The remaining [ ] are phase headers that auto-close when their sub-tasks are [x] (all are) — recommend marking 1/2/3/4/5 complete; keep 6 open until 6.2 lands.

Task 6.2 — Live 3-node failover end-to-end (the real PENDING work)

What is missing (vs. what is done):

• DONE in-env: 3-node deploy-ha.sh cluster elected a primary; killing it triggered server-side failover (ctrl-2 → term=2); the run REPRODUCED the original Invalid argument bug and the failover loop now ENGAGES (xcecli_joblist→xcecli_retry_next→xce_failover_rotate). Two live-only bugs found+fixed with regression unit tests: bug-609 (fresh-call-after-disconnect -EINVAL classified terminal → known-dead-ctx override in decide) and bug-610 (xce_failover_rotate SEGV — reconnect never re-bound ep->iomgr/handlers; fixed via new xce_ctx_reconnect()).
• MISSING: a clean, visually-captured within-session demonstration of the three acceptance outcomes against a stable runtime: (a) joblist reports cluster-not-ready during the election window then transparently recovers with NO EINVAL; (b) a request to a standby lands on the primary in one hop via the populated redirect hint (and via rotation when hint absent); (c) a jobsub spanning the failover creates exactly one job (idempotency end-to-end).

Integration points to exercise: /xce-deploy-ha localhost:9991 9992 9993 → xce_test (set masters, client, jobsub, joblist) → kill primary → re-joblist. NOTE the build hazard recorded twice in Notes: building lib removes the controller binary from the shared build dir — must xce-build.sh controller (relinks xce_test) before each deploy.

Options:

• Option A (recommended): stable-env manual E2E. Run the reproduction in a non-sandbox env where backgrounded controller procs survive between steps (the sandbox reaps them). Capture the 3 outcomes. Effort S–M, Risk Low — code already proven to engage; this is confirmation + evidence capture.
• Option B: scripted deterministic E2E harness (a testsuite shell/expect script that boots the cluster, kills the primary on a signal, asserts on xce_test output). Effort M, Risk Med — gives a repeatable regression gate but adds harness maintenance; process-lifecycle flakiness is the risk.
• Option C: in-process integration test simulating the controller side with fakes (no real procs). Effort M–L, Risk Med — avoids deploy flakiness but re-tests what unit tests already cover (low marginal value vs. real-cluster confidence); does NOT satisfy 6.2's "original reproduction" intent.

Recommendation: Option A to close 6.2 (intent = the original bug reproduction passes live), optionally backstopped by Option B as a CI regression gate in a follow-on.

Phase headers 1–5 (and 6) — administrative

Gap: checkbox-only. Action: flip 1/2/3/4/5 to [x] (all sub-tasks verified [x] above); leave 6 [ ] until 6.2 closes. Effort S, Risk None.

Research-Needed (for any remaining design/exec)

1. Stable runtime for live E2E — identify an env where backgrounded xce_controller procs persist across orchestration steps (sandbox reaps them; nix runtime env needed). Blocks 6.2-A.
2. Build-order guard — codify "rebuild controller after any lib build" (Notes 3.1/6.2 twice). Consider a xce-build.sh ordering note or a make dependency so xce_test is never stale-linked.
3. Idempotency end-to-end proof (5.2) — confirm the controller's client-jobid replay guard (job.c:56,95) actually returns E_XCE_DUP_JOB/-EEXIST on the re-sent write across a real failover (unit-mocked today; live single-job-created assertion is the open evidence).
4. Downstream readiness (out-of-boundary, flag only) — component-failover-adoption (HARD dep) still owes: per-daemon full master-list wiring (today even CLI uses masters[0], client.c:131) + liveness-safe bounded-blocking verification for async C daemons (worker controller_connector.c, scheduler) so backoff never stalls the heartbeat/iomgr thread. Not 6.2's scope, but it is the gate before this failover mechanism delivers value to real daemons.

Integration Risks (remaining)

• Process-lifecycle flakiness in any live/scripted E2E (controller procs, port reuse, election timing) — the dominant risk for 6.2; mitigated by Option A in a stable env + jittered election.
• Stale xce_test link if controller isn't rebuilt after a lib build — would mask a real fix or surface a phantom one. Mitigation = Research-Needed #2.
• Idempotency dependency (5.2) is external (controller job.c) — a regression there breaks exactly-once across failover without touching libxce. Mitigation = Research-Needed #3.