myxce/.kiro/specs/libxce-failover/tasks.md

Implementation Plan

• 1. Foundation: failover state, captured-reply fields, and cleanup

• 1.1 Establish shared failover state and reply-capture fields

  • Expose the client context to a new client-internal header so the failover helper can operate on it
  • Add per-context failover state: a lock, the current-master cursor, retry parameters (with safe defaults), and a reconnect/rejoin callback slot
  • Add fixed-size redirect-hint host/port fields to the client reply context (advisory hint; no sequence correlation)
  • Initialize all new state when the client starts; keep the retry-config defaults; remove the orphaned, never-wired retry primitive
  • Done: libxce builds with the new per-context state present and the dead primitive removed; retry defaults are reachable
  • Requirements: 6.1, 6.2

• 1.2 Realign the retry test harness to the new model (RED)

  • Retarget the existing libxce retry tests from the removed primitive to the new classifier and client-wrapper retry path
  • Add fixtures simulating each controller status (redirect with and without address, no-primary, not-ready, standby, duplicate-job) and transport failures, plus a fake multi-master list and a shared-context concurrency harness
  • Done: the test file compiles against the new symbols and fails (RED) because the implementation does not yet exist
  • Requirements: 2.5, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 8.3, 8.4

• 2. Core: status classification, endpoint rotation, reply-path capture

• 2.1 Implement status classification

  • Map each controller status and transport result to one of four actions: success, retry-now (redirect/standby/transport — rotate without backoff), retry-after-backoff (no-primary/not-ready), terminal
  • Never classify redirect, standby, no-primary, or not-ready as success
  • Treat a duplicate-job/exists status on a re-sent write as success (idempotent replay); on a first attempt treat it as a genuine duplicate surfaced to the caller
  • Done: classification unit tests pass for every status and for both duplicate-replay cases
  • Requirements: 2.5, 4.1, 4.2, 4.3, 4.5, 5.2
  • Boundary: StatusClassifier

• 2.2 Implement endpoint reconnect and master rotation

  • On rotate, try the captured redirect-hint address first when one is set (then clear it); otherwise advance the master cursor to the next configured master and reconnect
  • Close the stale connection first and reuse the same endpoint object so reply routing stays intact; a failed hint simply falls through to the next master
  • Done: rotation unit tests show hint-first then next-master fallback, reusing a single endpoint object, with correctness independent of the hint
  • Requirements: 2.1, 2.2, 2.3, 4.4, 8.4
  • Boundary: EndpointRotator
  • Depends: 1.1

• 2.3 (P) Capture redirect on the reply path and wake the wait on endpoint error

  • In the reply handler, copy the redirect-hint host/port into the client reply context when a redirect reply carries one, before the reply is freed; leave empty otherwise (advisory hint, no sequence correlation)
  • On endpoint error or peer reset, release any pending reply wait with a transport error so the in-flight call returns instead of hanging
  • Keep the status flowing through the existing wait-channel path so the public API layer stays unchanged
  • Done: after a redirect reply the captured hint address is readable from the context; a simulated endpoint error releases a pending wait with a transport error (no hang); the API layer is untouched
  • Requirements: 2.2, 4.4, 8.3
  • Boundary: RedirectCapture, ReplyWaitWake
  • Depends: 1.1

• 3. Core: the retry/failover decision helper

• 3.1 Implement the decision loop (classify, rotate, backoff, budget)

  • Combine classification, rotation, and the captured redirect into one decision: success and terminal stop the loop; redirect, standby, and transport results rotate without backoff; no-primary and not-ready apply jittered backoff
  • Track the retry budget; on exhaustion with no primary, return the distinct cluster-not-ready code
  • Update the sticky current-master on a successful served-as-primary reply; log the target on each rotation and the attempt count on budget exhaustion
  • Done: helper-level tests show no-primary-until-budget returns cluster-not-ready, become-primary returns success with the cursor advanced, and redirect/transport rotate without the no-primary backoff
  • Requirements: 1.2, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.4, 6.3, 7.1, 7.2
  • Depends: 2.1, 2.2, 2.3
  • Boundary: xcecli_retry_next

• 3.2 Implement reconnect recovery: context-state reset and stale-reply drain

  • After a successful reconnect, restore the context from its error state to its running state before the next attempt, so a transport error never leaves the context returning an invalid-argument error
  • Drain any stale reply left in the wait channel after reconnect so the next attempt cannot consume a prior-attempt reply
  • Done: a transport error followed by reconnect no longer surfaces an invalid-argument error and the context is reusable; a unit test injecting a stale reply confirms it is drained, not returned
  • Requirements: 1.2, 4.4
  • Depends: 3.1
  • Boundary: xcecli_retry_next

• 4. Core: reconnect/rejoin hook

• 4.1 Add the reconnect/rejoin hook mechanism

  • Provide a client API to register a reconnect callback and argument, stored in the failover state
  • Invoke the callback after each successful reconnect to a new primary and before resuming the interrupted op; treat a failing callback as a failed attempt that rotates or backs off per its result
  • Done: with a registered hook a simulated failover invokes it before resume; with no hook the loop resumes directly
  • Requirements: 1.2
  • Depends: 3.1, 3.2

• 5. Integration: wrap the client ops; controller redirect address

• 5.1 Wrap reply-bearing client ops in the failover loop under the per-context lock

  • Wrap each reply-bearing read and write op so it runs its single-shot call inside the decision loop, holding the per-context lock for the loop's duration
  • Tag write ops so the idempotent-replay rule applies; preserve every public signature and success/error convention
  • Done: existing callers compile unchanged; a reply-bearing op transparently completes against a new primary after a simulated failover; concurrent use of one context is serialized
  • Requirements: 1.1, 1.3, 2.4, 5.1, 5.3
  • Depends: 3.1, 3.2, 4.1
  • Boundary: XcecliRetryLoop, FailoverState

• 5.2 Apply best-effort reconnect to fire-and-forget ops

  • Give the notify, heartbeat, and wakeup ops a transport-reconnect-only variant: reconnect on a known-dead connection, with no status classification and no no-primary backoff
  • Done: a fire-and-forget op on a dead connection triggers a reconnect attempt and never blocks on the no-primary backoff budget; lost-notification redelivery remains out of scope
  • Requirements: 1.1, 5.3
  • Depends: 3.1
  • Boundary: XcecliRetryLoop

• 5.3 (P) Populate the primary address in controller redirect replies

  • On the standby redirect path, fill the reply's redirect host and port from the known primary address; keep returning the no-primary status when the primary is unknown
  • Audit for any other redirect-emitting sites and apply the same population
  • Add a controller unit test: the redirect path populates the address, and an unknown primary still yields the no-primary status
  • Done: the controller test passes and a standby redirect reply carries the primary address
  • Requirements: 8.1, 8.2
  • Boundary: ControllerRedirectAddress

• 6. Validation: unit suite and live failover

• 6.1 Bring the realigned unit suite green

  • Make the retry tests pass for classification, budget-to-cluster-not-ready, sticky cursor, hint-first then rotate, duplicate-replay-to-success, jitter and many-client staggering, stale-reply drain, context-state recovery, and concurrent shared-context use
  • Done: the full libxce retry unit suite passes under the project's test and ASAN build
  • Requirements: 2.4, 2.5, 3.2, 3.3, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 6.2, 6.3, 8.3, 8.4
  • Depends: 5.1, 5.2

• 6.2 Live 3-node failover end-to-end

  • On a 3-node cluster, submit jobs, kill the primary, and verify joblist reports cluster-not-ready during the election window then transparently succeeds against the new primary
  • Verify a request sent to a standby lands on the primary — in one hop when the redirect hint is present, otherwise via master-list rotation
  • Verify a job submission spanning a failover creates exactly one job
  • Done: the original reproduction passes — no invalid-argument error, transparent recovery, and a single job created
  • Requirements: 1.2, 3.2, 3.3, 5.1, 5.2, 8.1, 8.3
  • Depends: 5.1, 5.3

Implementation Notes

• 1.1: design said XCE_MAXHOST_LEN; real constant is XCE_MAXHOSTNAME_LEN (config.h, =256) — design corrected. api_ctx_t now in private client_internal.h; xcecli_reconnect_cb typedef lives there until task 4.1 adds the public setter. Build: cd src && nix develop --command bash tools/xce-build.sh lib.