106 lines
10 KiB
Markdown
106 lines
10 KiB
Markdown
# Implementation Plan
|
|
|
|
- [ ] 1. Foundation: failover state, captured-reply fields, and cleanup
|
|
- [x] 1.1 Establish shared failover state and reply-capture fields
|
|
- Expose the client context to a new client-internal header so the failover helper can operate on it
|
|
- Add per-context failover state: a lock, the current-master cursor, retry parameters (with safe defaults), and a reconnect/rejoin callback slot
|
|
- Add fixed-size redirect-hint host/port fields to the client reply context (advisory hint; no sequence correlation)
|
|
- Initialize all new state when the client starts; keep the retry-config defaults; remove the orphaned, never-wired retry primitive
|
|
- Done: libxce builds with the new per-context state present and the dead primitive removed; retry defaults are reachable
|
|
- _Requirements: 6.1, 6.2_
|
|
- [x] 1.2 Realign the retry test harness to the new model (RED)
|
|
- Retarget the existing libxce retry tests from the removed primitive to the new classifier and client-wrapper retry path
|
|
- Add fixtures simulating each controller status (redirect with and without address, no-primary, not-ready, standby, duplicate-job) and transport failures, plus a fake multi-master list and a shared-context concurrency harness
|
|
- Done: the test file compiles against the new symbols and fails (RED) because the implementation does not yet exist
|
|
- _Requirements: 2.5, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 8.3, 8.4_
|
|
|
|
- [ ] 2. Core: status classification, endpoint rotation, reply-path capture
|
|
- [x] 2.1 Implement status classification
|
|
- Map each controller status and transport result to one of four actions: success, retry-now (redirect/standby/transport — rotate without backoff), retry-after-backoff (no-primary/not-ready), terminal
|
|
- Never classify redirect, standby, no-primary, or not-ready as success
|
|
- Treat a duplicate-job/exists status on a re-sent write as success (idempotent replay); on a first attempt treat it as a genuine duplicate surfaced to the caller
|
|
- Done: classification unit tests pass for every status and for both duplicate-replay cases
|
|
- _Requirements: 2.5, 4.1, 4.2, 4.3, 4.5, 5.2_
|
|
- _Boundary: StatusClassifier_
|
|
- [x] 2.2 Implement endpoint reconnect and master rotation
|
|
- On rotate, try the captured redirect-hint address first when one is set (then clear it); otherwise advance the master cursor to the next configured master and reconnect
|
|
- Close the stale connection first and reuse the same endpoint object so reply routing stays intact; a failed hint simply falls through to the next master
|
|
- Done: rotation unit tests show hint-first then next-master fallback, reusing a single endpoint object, with correctness independent of the hint
|
|
- _Requirements: 2.1, 2.2, 2.3, 4.4, 8.4_
|
|
- _Boundary: EndpointRotator_
|
|
- _Depends: 1.1_
|
|
- [x] 2.3 (P) Capture redirect on the reply path and wake the wait on endpoint error
|
|
- In the reply handler, copy the redirect-hint host/port into the client reply context when a redirect reply carries one, before the reply is freed; leave empty otherwise (advisory hint, no sequence correlation)
|
|
- On endpoint error or peer reset, release any pending reply wait with a transport error so the in-flight call returns instead of hanging
|
|
- Keep the status flowing through the existing wait-channel path so the public API layer stays unchanged
|
|
- Done: after a redirect reply the captured hint address is readable from the context; a simulated endpoint error releases a pending wait with a transport error (no hang); the API layer is untouched
|
|
- _Requirements: 2.2, 4.4, 8.3_
|
|
- _Boundary: RedirectCapture, ReplyWaitWake_
|
|
- _Depends: 1.1_
|
|
|
|
- [ ] 3. Core: the retry/failover decision helper
|
|
- [ ] 3.1 Implement the decision loop (classify, rotate, backoff, budget)
|
|
- Combine classification, rotation, and the captured redirect into one decision: success and terminal stop the loop; redirect, standby, and transport results rotate without backoff; no-primary and not-ready apply jittered backoff
|
|
- Track the retry budget; on exhaustion with no primary, return the distinct cluster-not-ready code
|
|
- Update the sticky current-master on a successful served-as-primary reply; log the target on each rotation and the attempt count on budget exhaustion
|
|
- Done: helper-level tests show no-primary-until-budget returns cluster-not-ready, become-primary returns success with the cursor advanced, and redirect/transport rotate without the no-primary backoff
|
|
- _Requirements: 1.2, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.4, 6.3, 7.1, 7.2_
|
|
- _Depends: 2.1, 2.2, 2.3_
|
|
- _Boundary: xcecli_retry_next_
|
|
- [ ] 3.2 Implement reconnect recovery: context-state reset and stale-reply drain
|
|
- After a successful reconnect, restore the context from its error state to its running state before the next attempt, so a transport error never leaves the context returning an invalid-argument error
|
|
- Drain any stale reply left in the wait channel after reconnect so the next attempt cannot consume a prior-attempt reply
|
|
- Done: a transport error followed by reconnect no longer surfaces an invalid-argument error and the context is reusable; a unit test injecting a stale reply confirms it is drained, not returned
|
|
- _Requirements: 1.2, 4.4_
|
|
- _Depends: 3.1_
|
|
- _Boundary: xcecli_retry_next_
|
|
|
|
- [ ] 4. Core: reconnect/rejoin hook
|
|
- [ ] 4.1 Add the reconnect/rejoin hook mechanism
|
|
- Provide a client API to register a reconnect callback and argument, stored in the failover state
|
|
- Invoke the callback after each successful reconnect to a new primary and before resuming the interrupted op; treat a failing callback as a failed attempt that rotates or backs off per its result
|
|
- Done: with a registered hook a simulated failover invokes it before resume; with no hook the loop resumes directly
|
|
- _Requirements: 1.2_
|
|
- _Depends: 3.1, 3.2_
|
|
|
|
- [ ] 5. Integration: wrap the client ops; controller redirect address
|
|
- [ ] 5.1 Wrap reply-bearing client ops in the failover loop under the per-context lock
|
|
- Wrap each reply-bearing read and write op so it runs its single-shot call inside the decision loop, holding the per-context lock for the loop's duration
|
|
- Tag write ops so the idempotent-replay rule applies; preserve every public signature and success/error convention
|
|
- Done: existing callers compile unchanged; a reply-bearing op transparently completes against a new primary after a simulated failover; concurrent use of one context is serialized
|
|
- _Requirements: 1.1, 1.3, 2.4, 5.1, 5.3_
|
|
- _Depends: 3.1, 3.2, 4.1_
|
|
- _Boundary: XcecliRetryLoop, FailoverState_
|
|
- [ ] 5.2 Apply best-effort reconnect to fire-and-forget ops
|
|
- Give the notify, heartbeat, and wakeup ops a transport-reconnect-only variant: reconnect on a known-dead connection, with no status classification and no no-primary backoff
|
|
- Done: a fire-and-forget op on a dead connection triggers a reconnect attempt and never blocks on the no-primary backoff budget; lost-notification redelivery remains out of scope
|
|
- _Requirements: 1.1, 5.3_
|
|
- _Depends: 3.1_
|
|
- _Boundary: XcecliRetryLoop_
|
|
- [ ] 5.3 (P) Populate the primary address in controller redirect replies
|
|
- On the standby redirect path, fill the reply's redirect host and port from the known primary address; keep returning the no-primary status when the primary is unknown
|
|
- Audit for any other redirect-emitting sites and apply the same population
|
|
- Add a controller unit test: the redirect path populates the address, and an unknown primary still yields the no-primary status
|
|
- Done: the controller test passes and a standby redirect reply carries the primary address
|
|
- _Requirements: 8.1, 8.2_
|
|
- _Boundary: ControllerRedirectAddress_
|
|
|
|
- [ ] 6. Validation: unit suite and live failover
|
|
- [ ] 6.1 Bring the realigned unit suite green
|
|
- Make the retry tests pass for classification, budget-to-cluster-not-ready, sticky cursor, hint-first then rotate, duplicate-replay-to-success, jitter and many-client staggering, stale-reply drain, context-state recovery, and concurrent shared-context use
|
|
- Done: the full libxce retry unit suite passes under the project's test and ASAN build
|
|
- _Requirements: 2.4, 2.5, 3.2, 3.3, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 6.2, 6.3, 8.3, 8.4_
|
|
- _Depends: 5.1, 5.2_
|
|
- [ ] 6.2 Live 3-node failover end-to-end
|
|
- On a 3-node cluster, submit jobs, kill the primary, and verify joblist reports cluster-not-ready during the election window then transparently succeeds against the new primary
|
|
- Verify a request sent to a standby lands on the primary — in one hop when the redirect hint is present, otherwise via master-list rotation
|
|
- Verify a job submission spanning a failover creates exactly one job
|
|
- Done: the original reproduction passes — no invalid-argument error, transparent recovery, and a single job created
|
|
- _Requirements: 1.2, 3.2, 3.3, 5.1, 5.2, 8.1, 8.3_
|
|
- _Depends: 5.1, 5.3_
|
|
|
|
## Implementation Notes
|
|
- 1.1: design said `XCE_MAXHOST_LEN`; real constant is `XCE_MAXHOSTNAME_LEN` (config.h, =256) — design corrected. `api_ctx_t` now in private `client_internal.h`; `xcecli_reconnect_cb` typedef lives there until task 4.1 adds the public setter. Build: `cd src && nix develop --command bash tools/xce-build.sh lib`.
|
|
- 1.2 (ADAPTED): RED-against-undefined-symbols would break linking of the shared lib test binary across tasks, so 1.2 builds a GREEN reusable harness (`src/lib/testsuite/failover_harness.h`): fake multi-master config+cursor, an 11-row controller-status taxonomy table (resolves to real errnox.h codes), and an N-pthread runner. Component RED→GREEN tests live in 2.1/2.2/2.3/3.1/3.2. The harness defines `fo_expected_action_t` (FO_ACT_*) mirroring the design `xce_fo_action_t` — 2.1 asserts the real classifier against these rows.
|
|
- 2.2: EndpointRotator `xce_failover_next_target` (pure, socket-free: hint-first/consume, else cursor advance+wrap over conf->masters[]) + thin `xce_failover_rotate` (close-old/reconnect same endpoint). Live connect path is E2E (6.2). LAYERING: functions taking the private `api_ctx_t` are declared in `src/lib/client_internal.h`, NOT the public `xce/client_failover.h` (which keeps only the pure int-typed classifier) — per src/lib/CLAUDE.md (public headers in src/include/xce/ must not pull private state).
|
|
- 2.3: RedirectCapture via non-static seam `xce_ctx_capture_redirect_hint()` called in handle_reply BEFORE the capnp free AND before the `status<0` early return (so negative E_XCE_REDIRECT still captures). Bounded strncpy into redirect_hint_host. ReplyWaitWake needed NO production change — existing `handle_disconnect` (err_handler, xcectx.c) already injects -ECONNRESET into wbuf + sets XCE_CTX_ERR; full err_handler path is covered by E2E (6.2).
|