myxce/.kiro/specs/libxce-failover/tasks.md

Implementation Plan

• 1. Foundation: failover state, captured-reply fields, and cleanup

• 1.1 Establish shared failover state and reply-capture fields

  • Expose the client context to a new client-internal header so the failover helper can operate on it
  • Add per-context failover state: a lock, the current-master cursor, retry parameters (with safe defaults), and a reconnect/rejoin callback slot
  • Add fixed-size redirect-hint host/port fields to the client reply context (advisory hint; no sequence correlation)
  • Initialize all new state when the client starts; keep the retry-config defaults; remove the orphaned, never-wired retry primitive
  • Done: libxce builds with the new per-context state present and the dead primitive removed; retry defaults are reachable
  • Requirements: 6.1, 6.2

• 1.2 Realign the retry test harness to the new model (RED)

  • Retarget the existing libxce retry tests from the removed primitive to the new classifier and client-wrapper retry path
  • Add fixtures simulating each controller status (redirect with and without address, no-primary, not-ready, standby, duplicate-job) and transport failures, plus a fake multi-master list and a shared-context concurrency harness
  • Done: the test file compiles against the new symbols and fails (RED) because the implementation does not yet exist
  • Requirements: 2.5, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 8.3, 8.4

• 2. Core: status classification, endpoint rotation, reply-path capture

• 2.1 Implement status classification

  • Map each controller status and transport result to one of four actions: success, retry-now (redirect/standby/transport — rotate without backoff), retry-after-backoff (no-primary/not-ready), terminal
  • Never classify redirect, standby, no-primary, or not-ready as success
  • Treat a duplicate-job/exists status on a re-sent write as success (idempotent replay); on a first attempt treat it as a genuine duplicate surfaced to the caller
  • Done: classification unit tests pass for every status and for both duplicate-replay cases
  • Requirements: 2.5, 4.1, 4.2, 4.3, 4.5, 5.2
  • Boundary: StatusClassifier

• 2.2 Implement endpoint reconnect and master rotation

  • On rotate, try the captured redirect-hint address first when one is set (then clear it); otherwise advance the master cursor to the next configured master and reconnect
  • Close the stale connection first and reuse the same endpoint object so reply routing stays intact; a failed hint simply falls through to the next master
  • Done: rotation unit tests show hint-first then next-master fallback, reusing a single endpoint object, with correctness independent of the hint
  • Requirements: 2.1, 2.2, 2.3, 4.4, 8.4
  • Boundary: EndpointRotator
  • Depends: 1.1

• 2.3 (P) Capture redirect on the reply path and wake the wait on endpoint error

  • In the reply handler, copy the redirect-hint host/port into the client reply context when a redirect reply carries one, before the reply is freed; leave empty otherwise (advisory hint, no sequence correlation)
  • On endpoint error or peer reset, release any pending reply wait with a transport error so the in-flight call returns instead of hanging
  • Keep the status flowing through the existing wait-channel path so the public API layer stays unchanged
  • Done: after a redirect reply the captured hint address is readable from the context; a simulated endpoint error releases a pending wait with a transport error (no hang); the API layer is untouched
  • Requirements: 2.2, 4.4, 8.3
  • Boundary: RedirectCapture, ReplyWaitWake
  • Depends: 1.1

• 3. Core: the retry/failover decision helper

• 3.1 Implement the decision loop (classify, rotate, backoff, budget)

  • Combine classification, rotation, and the captured redirect into one decision: success and terminal stop the loop; redirect, standby, and transport results rotate without backoff; no-primary and not-ready apply jittered backoff
  • Track the retry budget; on exhaustion with no primary, return the distinct cluster-not-ready code
  • Update the sticky current-master on a successful served-as-primary reply; log the target on each rotation and the attempt count on budget exhaustion
  • Done: helper-level tests show no-primary-until-budget returns cluster-not-ready, become-primary returns success with the cursor advanced, and redirect/transport rotate without the no-primary backoff
  • Requirements: 1.2, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.4, 6.3, 7.1, 7.2
  • Depends: 2.1, 2.2, 2.3
  • Boundary: xcecli_retry_next

• 3.2 Implement reconnect recovery: context-state reset and stale-reply drain

  • After a successful reconnect, restore the context from its error state to its running state before the next attempt, so a transport error never leaves the context returning an invalid-argument error
  • Drain any stale reply left in the wait channel after reconnect so the next attempt cannot consume a prior-attempt reply
  • Done: a transport error followed by reconnect no longer surfaces an invalid-argument error and the context is reusable; a unit test injecting a stale reply confirms it is drained, not returned
  • Requirements: 1.2, 4.4
  • Depends: 3.1
  • Boundary: xcecli_retry_next

• 4. Core: reconnect/rejoin hook

• 4.1 Add the reconnect/rejoin hook mechanism

  • Provide a client API to register a reconnect callback and argument, stored in the failover state
  • Invoke the callback after each successful reconnect to a new primary and before resuming the interrupted op; treat a failing callback as a failed attempt that rotates or backs off per its result
  • Done: with a registered hook a simulated failover invokes it before resume; with no hook the loop resumes directly
  • Requirements: 1.2
  • Depends: 3.1, 3.2

• 5. Integration: wrap the client ops; controller redirect address

• 5.1 Wrap reply-bearing client ops in the failover loop under the per-context lock

  • Wrap each reply-bearing read and write op so it runs its single-shot call inside the decision loop, holding the per-context lock for the loop's duration
  • Tag write ops so the idempotent-replay rule applies; preserve every public signature and success/error convention
  • Done: existing callers compile unchanged; a reply-bearing op transparently completes against a new primary after a simulated failover; concurrent use of one context is serialized
  • Requirements: 1.1, 1.3, 2.4, 5.1, 5.3
  • Depends: 3.1, 3.2, 4.1
  • Boundary: XcecliRetryLoop, FailoverState

• 5.2 Apply best-effort reconnect to fire-and-forget ops

  • Give the notify, heartbeat, and wakeup ops a transport-reconnect-only variant: reconnect on a known-dead connection, with no status classification and no no-primary backoff
  • Done: a fire-and-forget op on a dead connection triggers a reconnect attempt and never blocks on the no-primary backoff budget; lost-notification redelivery remains out of scope
  • Requirements: 1.1, 5.3
  • Depends: 3.1
  • Boundary: XcecliRetryLoop

• 5.3 (P) Populate the primary address in controller redirect replies

  • On the standby redirect path, fill the reply's redirect host and port from the known primary address; keep returning the no-primary status when the primary is unknown
  • Audit for any other redirect-emitting sites and apply the same population
  • Add a controller unit test: the redirect path populates the address, and an unknown primary still yields the no-primary status
  • Done: the controller test passes and a standby redirect reply carries the primary address
  • Requirements: 8.1, 8.2
  • Boundary: ControllerRedirectAddress

• 6. Validation: unit suite and live failover

• 6.1 Bring the realigned unit suite green

  • Make the retry tests pass for classification, budget-to-cluster-not-ready, sticky cursor, hint-first then rotate, duplicate-replay-to-success, jitter and many-client staggering, stale-reply drain, context-state recovery, and concurrent shared-context use
  • Done: the full libxce retry unit suite passes under the project's test and ASAN build
  • Requirements: 2.4, 2.5, 3.2, 3.3, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 6.2, 6.3, 8.3, 8.4
  • Depends: 5.1, 5.2

• 6.2 Live 3-node failover end-to-end

  • On a 3-node cluster, submit jobs, kill the primary, and verify joblist reports cluster-not-ready during the election window then transparently succeeds against the new primary
  • Verify a request sent to a standby lands on the primary — in one hop when the redirect hint is present, otherwise via master-list rotation
  • Verify a job submission spanning a failover creates exactly one job
  • Done: the original reproduction passes — no invalid-argument error, transparent recovery, and a single job created
  • Requirements: 1.2, 3.2, 3.3, 5.1, 5.2, 8.1, 8.3
  • Depends: 5.1, 5.3

Implementation Notes

• 6.2 VERIFIED 2026-07-02: deployed a 3-node cluster to ~/tmp/xce-failover-e2e-deploy (real ext4). Raced a jobsub call (masters=all 3) against a background kill -9 of the primary (ctrl-1, term 2) fired 0.4s in — the call spanned the primary's death and returned jobsubmit ok with no error surfaced to the caller; ctrl-3 was promoted (term 2→4). Post-failover expect-jobs 1 confirmed exactly ONE job exists — no duplicate from the in-flight retry. Standby-lands-on-primary verified via master-list rotation (masters=all 3 transparently found the new primary via wait-primary); the redirect-hint path (E_XCE_REDIRECT w/ address) is specific to OPCode_join (join.c fill_redirect_reply) and not exercised by job ops, which instead get E_XCE_STANDBY (classified XCE_FO_RETRY_NOW, rotates via master list) — confirmed by a control probe: pointing masters at ONLY a standby with no other candidates correctly surfaces the standby error (nowhere to rotate to), matching the classifier's documented Invariant 2.5. PASS on all 3 Observable bullets and the Done criteria.

Implementation Notes

• 1.1: design said XCE_MAXHOST_LEN; real constant is XCE_MAXHOSTNAME_LEN (config.h, =256) — design corrected. api_ctx_t now in private client_internal.h; xcecli_reconnect_cb typedef lives there until task 4.1 adds the public setter. Build: cd src && nix develop --command bash tools/xce-build.sh lib.
• 1.2 (ADAPTED): RED-against-undefined-symbols would break linking of the shared lib test binary across tasks, so 1.2 builds a GREEN reusable harness (src/lib/testsuite/failover_harness.h): fake multi-master config+cursor, an 11-row controller-status taxonomy table (resolves to real errnox.h codes), and an N-pthread runner. Component RED→GREEN tests live in 2.1/2.2/2.3/3.1/3.2. The harness defines fo_expected_action_t (FO_ACT_*) mirroring the design xce_fo_action_t — 2.1 asserts the real classifier against these rows.
• 2.2: EndpointRotator xce_failover_next_target (pure, socket-free: hint-first/consume, else cursor advance+wrap over conf->masters[]) + thin xce_failover_rotate (close-old/reconnect same endpoint). Live connect path is E2E (6.2). LAYERING: functions taking the private api_ctx_t are declared in src/lib/client_internal.h, NOT the public xce/client_failover.h (which keeps only the pure int-typed classifier) — per src/lib/CLAUDE.md (public headers in src/include/xce/ must not pull private state).
• 2.3: RedirectCapture via non-static seam xce_ctx_capture_redirect_hint() called in handle_reply BEFORE the capnp free AND before the status<0 early return (so negative E_XCE_REDIRECT still captures). Bounded strncpy into redirect_hint_host. ReplyWaitWake needed NO production change — existing handle_disconnect (err_handler, xcectx.c) already injects -ECONNRESET into wbuf + sets XCE_CTX_ERR; full err_handler path is covered by E2E (6.2).
• 3.1: pure xcecli_retry_decide (no I/O — classify+budget+jittered-backoff calc, D6 exhaustion: no-primary→-E_XCE_NOT_READY, transport→keep last rc) + thin xcecli_retry_next (decide + RETRY_BACKOFF nanosleep + xce_failover_rotate + tlog_info/tlog_warn). Logger = utils.h tlog_* macros. State-reset/drain deferred to 3.2; rejoin hook to 4.1.
• 3.2: xcecli_ctx_recover (socket-free): flips ac->ctx.state ERR→RUN ONLY, drains stale wbuf via non-blocking cbuf_tryget (-EAGAIN on empty); wired into xcecli_retry_next after successful rotate, before return true. THIS is the -EINVAL bug fix. Added missing prototype extern int cbuf_tryget(cbuf_t*,void*) to mylock.h (function already defined mylock.c:166, just undeclared).
• 4.1: public xcecli_reconnect_cb typedef + xcecli_set_reconnect_handler (client.h/client.c, stores cb/arg in failover); socket-free xcecli_invoke_rejoin wired into xcecli_retry_next AFTER rotate+recover, BEFORE resume. hook<0 = failed attempt → bounded tail-recursion (consumes attempt budget, capped at max_retries). typedef MOVED from client_internal.h to client.h. API additive.
• 5.1: XCECLI_FAILOVER_CALL(ac,is_write,EXPR) macro (client_internal.h) = lock(failover.lock) → retry init → do{rc=EXPR}while(xcecli_retry_next) → unlock → yield rc. Wrapped 18 reply-bearing ops; writes=true for jobsubmit/jobdispatch/jobctrl/queuectrl/hostctrl/update_resvals/update_jusage/runtask/join/set_pend_reason/clear_reason, reads=false otherwise. R1.3: zero signature changes. DESIGN CORRECTION: xce_nop is fire-and-forget (no cbuf_get, xceapi.c:30) — moved nop reply-read→fire-and-forget in design OpCategory; nop wraps in 5.2 not 5.1.
• 5.2: XCECLI_FF_CALL macro (single-shot, NO retry loop/backoff/budget): lock → if known-dead (xcecli_ff_needs_reconnect = ctx.state==XCE_CTX_ERR) one rotate+recover → one send → unlock. Applied at the 3 wire chokepoints xcecli_notify/heartbeat/nop (notify_* delegate to xcecli_notify). wakeup/try_wakeup are LOCAL-only (cbuf_tryput own wbuf, no wire send) so correctly NOT wrapped.
• 5.3: fill_redirect_reply(xce_reply_t*, const ha_state_t*) (non-static, join.c) sets redirect_host=ha.primary_host + redirect_port=(uint16_t)ha.primary_port; called ONLY on the E_XCE_REDIRECT branch before xce_reply. Audit: join.c is the SOLE E_XCE_REDIRECT emitter. NOMASTER path unchanged (8.2). host alias (no copy) safe: ha lives cluster-lifetime, xce_reply serializes synchronously. New test_join_redirect.c registered in controller_test.c + meson.build.
• 6.1: full lib suite green (161 tests). All 14 reqs (2.4,2.5,3.2,3.3,4.1-4.5,5.2,6.2,6.3,8.3,8.4) map to genuine passing tests. Gap-filled 6.3 with test_decide_jitter_staggers_many_clients (64 distinct-seed carriers → staggered delays, anti-thundering-herd). No production bugs found.
• 6.2 (LIVE E2E — partial): Live 3-node cluster (deploy-ha.sh) was brought up and DID elect a primary; killing the primary triggered server-side failover (ctrl-2 self-elected term=2). The run REPRODUCED the original Invalid argument client bug → root-caused + FIXED (bug-609, known-dead-ctx override) + regression unit test added. The clean within-session client transparent-recovery could NOT be visually demonstrated in this sandbox (background controller processes are reaped between tool calls; nix runtime env needed for backgrounded procs; lib build cleans the controller binary). RECOMMEND final visual confirmation in a stable env: build all, /xce-deploy-ha localhost:9991 localhost:9992 localhost:9993, run xce_test (set masters / client / jobsub / joblist), kill the primary, confirm joblist reports cluster-not-ready then transparently recovers (no EINVAL).
• 6.2 (LIVE, round 2 — your env): failover now ENGAGES (xcecli_joblist→xcecli_retry_next→xce_failover_rotate). Two live-only bugs found+fixed: bug-609 (fresh-call-after-disconnect -EINVAL classified terminal → known-dead-ctx override in decide) and bug-610 (xce_failover_rotate SEGV in endpoint_connect — reconnect never re-bound ep->iomgr nor re-armed owner/msg/err handlers + iomgr_monitor; fixed via new xce_ctx_reconnect() in xcectx.c that mirrors xcecli_start establishment without restarting iomgr/vtaskmgr or touching ctx->state). REBUILD controller before retest: cd src && nix develop --command bash tools/xce-build.sh controller (relinks xce_test). NOTE: building lib removes the controller binary from the shared build dir — always rebuild controller before deploy.