lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/libxce-failover/tasks.md

13 KiB
Raw Blame History

Implementation Plan

  • 1. Foundation: failover state, captured-reply fields, and cleanup

  • 1.1 Establish shared failover state and reply-capture fields

    • Expose the client context to a new client-internal header so the failover helper can operate on it
    • Add per-context failover state: a lock, the current-master cursor, retry parameters (with safe defaults), and a reconnect/rejoin callback slot
    • Add fixed-size redirect-hint host/port fields to the client reply context (advisory hint; no sequence correlation)
    • Initialize all new state when the client starts; keep the retry-config defaults; remove the orphaned, never-wired retry primitive
    • Done: libxce builds with the new per-context state present and the dead primitive removed; retry defaults are reachable
    • Requirements: 6.1, 6.2

  • 1.2 Realign the retry test harness to the new model (RED)

    • Retarget the existing libxce retry tests from the removed primitive to the new classifier and client-wrapper retry path
    • Add fixtures simulating each controller status (redirect with and without address, no-primary, not-ready, standby, duplicate-job) and transport failures, plus a fake multi-master list and a shared-context concurrency harness
    • Done: the test file compiles against the new symbols and fails (RED) because the implementation does not yet exist
    • Requirements: 2.5, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 8.3, 8.4

  • 2. Core: status classification, endpoint rotation, reply-path capture

  • 2.1 Implement status classification

    • Map each controller status and transport result to one of four actions: success, retry-now (redirect/standby/transport — rotate without backoff), retry-after-backoff (no-primary/not-ready), terminal
    • Never classify redirect, standby, no-primary, or not-ready as success
    • Treat a duplicate-job/exists status on a re-sent write as success (idempotent replay); on a first attempt treat it as a genuine duplicate surfaced to the caller
    • Done: classification unit tests pass for every status and for both duplicate-replay cases
    • Requirements: 2.5, 4.1, 4.2, 4.3, 4.5, 5.2
    • Boundary: StatusClassifier

  • 2.2 Implement endpoint reconnect and master rotation

    • On rotate, try the captured redirect-hint address first when one is set (then clear it); otherwise advance the master cursor to the next configured master and reconnect
    • Close the stale connection first and reuse the same endpoint object so reply routing stays intact; a failed hint simply falls through to the next master
    • Done: rotation unit tests show hint-first then next-master fallback, reusing a single endpoint object, with correctness independent of the hint
    • Requirements: 2.1, 2.2, 2.3, 4.4, 8.4
    • Boundary: EndpointRotator
    • Depends: 1.1

  • 2.3 (P) Capture redirect on the reply path and wake the wait on endpoint error

    • In the reply handler, copy the redirect-hint host/port into the client reply context when a redirect reply carries one, before the reply is freed; leave empty otherwise (advisory hint, no sequence correlation)
    • On endpoint error or peer reset, release any pending reply wait with a transport error so the in-flight call returns instead of hanging
    • Keep the status flowing through the existing wait-channel path so the public API layer stays unchanged
    • Done: after a redirect reply the captured hint address is readable from the context; a simulated endpoint error releases a pending wait with a transport error (no hang); the API layer is untouched
    • Requirements: 2.2, 4.4, 8.3
    • Boundary: RedirectCapture, ReplyWaitWake
    • Depends: 1.1

  • 3. Core: the retry/failover decision helper

  • 3.1 Implement the decision loop (classify, rotate, backoff, budget)

    • Combine classification, rotation, and the captured redirect into one decision: success and terminal stop the loop; redirect, standby, and transport results rotate without backoff; no-primary and not-ready apply jittered backoff
    • Track the retry budget; on exhaustion with no primary, return the distinct cluster-not-ready code
    • Update the sticky current-master on a successful served-as-primary reply; log the target on each rotation and the attempt count on budget exhaustion
    • Done: helper-level tests show no-primary-until-budget returns cluster-not-ready, become-primary returns success with the cursor advanced, and redirect/transport rotate without the no-primary backoff
    • Requirements: 1.2, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.4, 6.3, 7.1, 7.2
    • Depends: 2.1, 2.2, 2.3
    • Boundary: xcecli_retry_next

  • 3.2 Implement reconnect recovery: context-state reset and stale-reply drain

    • After a successful reconnect, restore the context from its error state to its running state before the next attempt, so a transport error never leaves the context returning an invalid-argument error
    • Drain any stale reply left in the wait channel after reconnect so the next attempt cannot consume a prior-attempt reply
    • Done: a transport error followed by reconnect no longer surfaces an invalid-argument error and the context is reusable; a unit test injecting a stale reply confirms it is drained, not returned
    • Requirements: 1.2, 4.4
    • Depends: 3.1
    • Boundary: xcecli_retry_next

  • 4. Core: reconnect/rejoin hook

  • 4.1 Add the reconnect/rejoin hook mechanism

    • Provide a client API to register a reconnect callback and argument, stored in the failover state
    • Invoke the callback after each successful reconnect to a new primary and before resuming the interrupted op; treat a failing callback as a failed attempt that rotates or backs off per its result
    • Done: with a registered hook a simulated failover invokes it before resume; with no hook the loop resumes directly
    • Requirements: 1.2
    • Depends: 3.1, 3.2

  • 5. Integration: wrap the client ops; controller redirect address

  • 5.1 Wrap reply-bearing client ops in the failover loop under the per-context lock

    • Wrap each reply-bearing read and write op so it runs its single-shot call inside the decision loop, holding the per-context lock for the loop's duration
    • Tag write ops so the idempotent-replay rule applies; preserve every public signature and success/error convention
    • Done: existing callers compile unchanged; a reply-bearing op transparently completes against a new primary after a simulated failover; concurrent use of one context is serialized
    • Requirements: 1.1, 1.3, 2.4, 5.1, 5.3
    • Depends: 3.1, 3.2, 4.1
    • Boundary: XcecliRetryLoop, FailoverState

  • 5.2 Apply best-effort reconnect to fire-and-forget ops

    • Give the notify, heartbeat, and wakeup ops a transport-reconnect-only variant: reconnect on a known-dead connection, with no status classification and no no-primary backoff
    • Done: a fire-and-forget op on a dead connection triggers a reconnect attempt and never blocks on the no-primary backoff budget; lost-notification redelivery remains out of scope
    • Requirements: 1.1, 5.3
    • Depends: 3.1
    • Boundary: XcecliRetryLoop

  • 5.3 (P) Populate the primary address in controller redirect replies

    • On the standby redirect path, fill the reply's redirect host and port from the known primary address; keep returning the no-primary status when the primary is unknown
    • Audit for any other redirect-emitting sites and apply the same population
    • Add a controller unit test: the redirect path populates the address, and an unknown primary still yields the no-primary status
    • Done: the controller test passes and a standby redirect reply carries the primary address
    • Requirements: 8.1, 8.2
    • Boundary: ControllerRedirectAddress

  • 6. Validation: unit suite and live failover

  • 6.1 Bring the realigned unit suite green

    • Make the retry tests pass for classification, budget-to-cluster-not-ready, sticky cursor, hint-first then rotate, duplicate-replay-to-success, jitter and many-client staggering, stale-reply drain, context-state recovery, and concurrent shared-context use
    • Done: the full libxce retry unit suite passes under the project's test and ASAN build
    • Requirements: 2.4, 2.5, 3.2, 3.3, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 6.2, 6.3, 8.3, 8.4
    • Depends: 5.1, 5.2

  • 6.2 Live 3-node failover end-to-end

    • On a 3-node cluster, submit jobs, kill the primary, and verify joblist reports cluster-not-ready during the election window then transparently succeeds against the new primary
    • Verify a request sent to a standby lands on the primary — in one hop when the redirect hint is present, otherwise via master-list rotation
    • Verify a job submission spanning a failover creates exactly one job
    • Done: the original reproduction passes — no invalid-argument error, transparent recovery, and a single job created
    • Requirements: 1.2, 3.2, 3.3, 5.1, 5.2, 8.1, 8.3
    • Depends: 5.1, 5.3

Implementation Notes

  • 1.1: design said XCE_MAXHOST_LEN; real constant is XCE_MAXHOSTNAME_LEN (config.h, =256) — design corrected. api_ctx_t now in private client_internal.h; xcecli_reconnect_cb typedef lives there until task 4.1 adds the public setter. Build: cd src && nix develop --command bash tools/xce-build.sh lib.
  • 1.2 (ADAPTED): RED-against-undefined-symbols would break linking of the shared lib test binary across tasks, so 1.2 builds a GREEN reusable harness (src/lib/testsuite/failover_harness.h): fake multi-master config+cursor, an 11-row controller-status taxonomy table (resolves to real errnox.h codes), and an N-pthread runner. Component RED→GREEN tests live in 2.1/2.2/2.3/3.1/3.2. The harness defines fo_expected_action_t (FO_ACT_*) mirroring the design xce_fo_action_t — 2.1 asserts the real classifier against these rows.
  • 2.2: EndpointRotator xce_failover_next_target (pure, socket-free: hint-first/consume, else cursor advance+wrap over conf->masters[]) + thin xce_failover_rotate (close-old/reconnect same endpoint). Live connect path is E2E (6.2). LAYERING: functions taking the private api_ctx_t are declared in src/lib/client_internal.h, NOT the public xce/client_failover.h (which keeps only the pure int-typed classifier) — per src/lib/CLAUDE.md (public headers in src/include/xce/ must not pull private state).
  • 2.3: RedirectCapture via non-static seam xce_ctx_capture_redirect_hint() called in handle_reply BEFORE the capnp free AND before the status<0 early return (so negative E_XCE_REDIRECT still captures). Bounded strncpy into redirect_hint_host. ReplyWaitWake needed NO production change — existing handle_disconnect (err_handler, xcectx.c) already injects -ECONNRESET into wbuf + sets XCE_CTX_ERR; full err_handler path is covered by E2E (6.2).
  • 3.1: pure xcecli_retry_decide (no I/O — classify+budget+jittered-backoff calc, D6 exhaustion: no-primary→-E_XCE_NOT_READY, transport→keep last rc) + thin xcecli_retry_next (decide + RETRY_BACKOFF nanosleep + xce_failover_rotate + tlog_info/tlog_warn). Logger = utils.h tlog_* macros. State-reset/drain deferred to 3.2; rejoin hook to 4.1.
  • 3.2: xcecli_ctx_recover (socket-free): flips ac->ctx.state ERR→RUN ONLY, drains stale wbuf via non-blocking cbuf_tryget (-EAGAIN on empty); wired into xcecli_retry_next after successful rotate, before return true. THIS is the -EINVAL bug fix. Added missing prototype extern int cbuf_tryget(cbuf_t*,void*) to mylock.h (function already defined mylock.c:166, just undeclared).
  • 4.1: public xcecli_reconnect_cb typedef + xcecli_set_reconnect_handler (client.h/client.c, stores cb/arg in failover); socket-free xcecli_invoke_rejoin wired into xcecli_retry_next AFTER rotate+recover, BEFORE resume. hook<0 = failed attempt → bounded tail-recursion (consumes attempt budget, capped at max_retries). typedef MOVED from client_internal.h to client.h. API additive.
  • 5.1: XCECLI_FAILOVER_CALL(ac,is_write,EXPR) macro (client_internal.h) = lock(failover.lock) → retry init → do{rc=EXPR}while(xcecli_retry_next) → unlock → yield rc. Wrapped 18 reply-bearing ops; writes=true for jobsubmit/jobdispatch/jobctrl/queuectrl/hostctrl/update_resvals/update_jusage/runtask/join/set_pend_reason/clear_reason, reads=false otherwise. R1.3: zero signature changes. DESIGN CORRECTION: xce_nop is fire-and-forget (no cbuf_get, xceapi.c:30) — moved nop reply-read→fire-and-forget in design OpCategory; nop wraps in 5.2 not 5.1.
  • 5.2: XCECLI_FF_CALL macro (single-shot, NO retry loop/backoff/budget): lock → if known-dead (xcecli_ff_needs_reconnect = ctx.state==XCE_CTX_ERR) one rotate+recover → one send → unlock. Applied at the 3 wire chokepoints xcecli_notify/heartbeat/nop (notify_* delegate to xcecli_notify). wakeup/try_wakeup are LOCAL-only (cbuf_tryput own wbuf, no wire send) so correctly NOT wrapped.
  • 5.3: fill_redirect_reply(xce_reply_t*, const ha_state_t*) (non-static, join.c) sets redirect_host=ha.primary_host + redirect_port=(uint16_t)ha.primary_port; called ONLY on the E_XCE_REDIRECT branch before xce_reply. Audit: join.c is the SOLE E_XCE_REDIRECT emitter. NOMASTER path unchanged (8.2). host alias (no copy) safe: ha lives cluster-lifetime, xce_reply serializes synchronously. New test_join_redirect.c registered in controller_test.c + meson.build.