# Implementation Plan

- [x] 1. Foundation: failover state, captured-reply fields, and cleanup
- [x] 1.1 Establish shared failover state and reply-capture fields
  - Expose the client context to a new client-internal header so the failover helper can operate on it
  - Add per-context failover state: a lock, the current-master cursor, retry parameters (with safe defaults), and a reconnect/rejoin callback slot
  - Add fixed-size redirect-hint host/port fields to the client reply context (advisory hint; no sequence correlation)
  - Initialize all new state when the client starts; keep the retry-config defaults; remove the orphaned, never-wired retry primitive
  - Done: libxce builds with the new per-context state present and the dead primitive removed; retry defaults are reachable
  - _Requirements: 6.1, 6.2_
- [x] 1.2 Realign the retry test harness to the new model (RED)
  - Retarget the existing libxce retry tests from the removed primitive to the new classifier and client-wrapper retry path
  - Add fixtures simulating each controller status (redirect with and without address, no-primary, not-ready, standby, duplicate-job) and transport failures, plus a fake multi-master list and a shared-context concurrency harness
  - Done: the test file compiles against the new symbols and fails (RED) because the implementation does not yet exist
  - _Requirements: 2.5, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 8.3, 8.4_

- [x] 2. Core: status classification, endpoint rotation, reply-path capture
- [x] 2.1 Implement status classification
  - Map each controller status and transport result to one of four actions: success, retry-now (redirect/standby/transport — rotate without backoff), retry-after-backoff (no-primary/not-ready), terminal
  - Never classify redirect, standby, no-primary, or not-ready as success
  - Treat a duplicate-job/exists status on a re-sent write as success (idempotent replay); on a first attempt treat it as a genuine duplicate surfaced to the caller
  - Done: classification unit tests pass for every status and for both duplicate-replay cases
  - _Requirements: 2.5, 4.1, 4.2, 4.3, 4.5, 5.2_
  - _Boundary: StatusClassifier_
- [x] 2.2 Implement endpoint reconnect and master rotation
  - On rotate, try the captured redirect-hint address first when one is set (then clear it); otherwise advance the master cursor to the next configured master and reconnect
  - Close the stale connection first and reuse the same endpoint object so reply routing stays intact; a failed hint simply falls through to the next master
  - Done: rotation unit tests show hint-first then next-master fallback, reusing a single endpoint object, with correctness independent of the hint
  - _Requirements: 2.1, 2.2, 2.3, 4.4, 8.4_
  - _Boundary: EndpointRotator_
  - _Depends: 1.1_
- [x] 2.3 (P) Capture redirect on the reply path and wake the wait on endpoint error
  - In the reply handler, copy the redirect-hint host/port into the client reply context when a redirect reply carries one, before the reply is freed; leave empty otherwise (advisory hint, no sequence correlation)
  - On endpoint error or peer reset, release any pending reply wait with a transport error so the in-flight call returns instead of hanging
  - Keep the status flowing through the existing wait-channel path so the public API layer stays unchanged
  - Done: after a redirect reply the captured hint address is readable from the context; a simulated endpoint error releases a pending wait with a transport error (no hang); the API layer is untouched
  - _Requirements: 2.2, 4.4, 8.3_
  - _Boundary: RedirectCapture, ReplyWaitWake_
  - _Depends: 1.1_

- [x] 3. Core: the retry/failover decision helper
- [x] 3.1 Implement the decision loop (classify, rotate, backoff, budget)
  - Combine classification, rotation, and the captured redirect into one decision: success and terminal stop the loop; redirect, standby, and transport results rotate without backoff; no-primary and not-ready apply jittered backoff
  - Track the retry budget; on exhaustion with no primary, return the distinct cluster-not-ready code
  - Update the sticky current-master on a successful served-as-primary reply; log the target on each rotation and the attempt count on budget exhaustion
  - Done: helper-level tests show no-primary-until-budget returns cluster-not-ready, become-primary returns success with the cursor advanced, and redirect/transport rotate without the no-primary backoff
  - _Requirements: 1.2, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.4, 6.3, 7.1, 7.2_
  - _Depends: 2.1, 2.2, 2.3_
  - _Boundary: xcecli_retry_next_
- [x] 3.2 Implement reconnect recovery: context-state reset and stale-reply drain
  - After a successful reconnect, restore the context from its error state to its running state before the next attempt, so a transport error never leaves the context returning an invalid-argument error
  - Drain any stale reply left in the wait channel after reconnect so the next attempt cannot consume a prior-attempt reply
  - Done: a transport error followed by reconnect no longer surfaces an invalid-argument error and the context is reusable; a unit test injecting a stale reply confirms it is drained, not returned
  - _Requirements: 1.2, 4.4_
  - _Depends: 3.1_
  - _Boundary: xcecli_retry_next_

- [x] 4. Core: reconnect/rejoin hook
- [x] 4.1 Add the reconnect/rejoin hook mechanism
  - Provide a client API to register a reconnect callback and argument, stored in the failover state
  - Invoke the callback after each successful reconnect to a new primary and before resuming the interrupted op; treat a failing callback as a failed attempt that rotates or backs off per its result
  - Done: with a registered hook a simulated failover invokes it before resume; with no hook the loop resumes directly
  - _Requirements: 1.2_
  - _Depends: 3.1, 3.2_

- [x] 5. Integration: wrap the client ops; controller redirect address
- [x] 5.1 Wrap reply-bearing client ops in the failover loop under the per-context lock
  - Wrap each reply-bearing read and write op so it runs its single-shot call inside the decision loop, holding the per-context lock for the loop's duration
  - Tag write ops so the idempotent-replay rule applies; preserve every public signature and success/error convention
  - Done: existing callers compile unchanged; a reply-bearing op transparently completes against a new primary after a simulated failover; concurrent use of one context is serialized
  - _Requirements: 1.1, 1.3, 2.4, 5.1, 5.3_
  - _Depends: 3.1, 3.2, 4.1_
  - _Boundary: XcecliRetryLoop, FailoverState_
- [x] 5.2 Apply best-effort reconnect to fire-and-forget ops
  - Give the notify, heartbeat, and wakeup ops a transport-reconnect-only variant: reconnect on a known-dead connection, with no status classification and no no-primary backoff
  - Done: a fire-and-forget op on a dead connection triggers a reconnect attempt and never blocks on the no-primary backoff budget; lost-notification redelivery remains out of scope
  - _Requirements: 1.1, 5.3_
  - _Depends: 3.1_
  - _Boundary: XcecliRetryLoop_
- [x] 5.3 (P) Populate the primary address in controller redirect replies
  - On the standby redirect path, fill the reply's redirect host and port from the known primary address; keep returning the no-primary status when the primary is unknown
  - Audit for any other redirect-emitting sites and apply the same population
  - Add a controller unit test: the redirect path populates the address, and an unknown primary still yields the no-primary status
  - Done: the controller test passes and a standby redirect reply carries the primary address
  - _Requirements: 8.1, 8.2_
  - _Boundary: ControllerRedirectAddress_

- [x] 6. Validation: unit suite and live failover
- [x] 6.1 Bring the realigned unit suite green
  - Make the retry tests pass for classification, budget-to-cluster-not-ready, sticky cursor, hint-first then rotate, duplicate-replay-to-success, jitter and many-client staggering, stale-reply drain, context-state recovery, and concurrent shared-context use
  - Done: the full libxce retry unit suite passes under the project's test and ASAN build
  - _Requirements: 2.4, 2.5, 3.2, 3.3, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 6.2, 6.3, 8.3, 8.4_
  - _Depends: 5.1, 5.2_
- [x] 6.2 Live 3-node failover end-to-end  <!-- VERIFIED 2026-07-02 on ~/tmp (ext4) — see Implementation Notes -->
  - On a 3-node cluster, submit jobs, kill the primary, and verify joblist reports cluster-not-ready during the election window then transparently succeeds against the new primary
  - Verify a request sent to a standby lands on the primary — in one hop when the redirect hint is present, otherwise via master-list rotation
  - Verify a job submission spanning a failover creates exactly one job
  - Done: the original reproduction passes — no invalid-argument error, transparent recovery, and a single job created
  - _Requirements: 1.2, 3.2, 3.3, 5.1, 5.2, 8.1, 8.3_
  - _Depends: 5.1, 5.3_

## Implementation Notes
- 6.2 VERIFIED 2026-07-02: deployed a 3-node cluster to `~/tmp/xce-failover-e2e-deploy` (real ext4). Raced a `jobsub` call (masters=all 3) against a background `kill -9` of the primary (ctrl-1, term 2) fired 0.4s in — the call spanned the primary's death and returned `jobsubmit ok` with no error surfaced to the caller; ctrl-3 was promoted (term 2→4). Post-failover `expect-jobs 1` confirmed exactly ONE job exists — no duplicate from the in-flight retry. Standby-lands-on-primary verified via master-list rotation (masters=all 3 transparently found the new primary via `wait-primary`); the redirect-hint path (`E_XCE_REDIRECT` w/ address) is specific to `OPCode_join` (join.c `fill_redirect_reply`) and not exercised by job ops, which instead get `E_XCE_STANDBY` (classified `XCE_FO_RETRY_NOW`, rotates via master list) — confirmed by a control probe: pointing masters at ONLY a standby with no other candidates correctly surfaces the standby error (nowhere to rotate to), matching the classifier's documented Invariant 2.5. PASS on all 3 Observable bullets and the Done criteria.

## Implementation Notes
- 1.1: design said `XCE_MAXHOST_LEN`; real constant is `XCE_MAXHOSTNAME_LEN` (config.h, =256) — design corrected. `api_ctx_t` now in private `client_internal.h`; `xcecli_reconnect_cb` typedef lives there until task 4.1 adds the public setter. Build: `cd src && nix develop --command bash tools/xce-build.sh lib`.
- 1.2 (ADAPTED): RED-against-undefined-symbols would break linking of the shared lib test binary across tasks, so 1.2 builds a GREEN reusable harness (`src/lib/testsuite/failover_harness.h`): fake multi-master config+cursor, an 11-row controller-status taxonomy table (resolves to real errnox.h codes), and an N-pthread runner. Component RED→GREEN tests live in 2.1/2.2/2.3/3.1/3.2. The harness defines `fo_expected_action_t` (FO_ACT_*) mirroring the design `xce_fo_action_t` — 2.1 asserts the real classifier against these rows.
- 2.2: EndpointRotator `xce_failover_next_target` (pure, socket-free: hint-first/consume, else cursor advance+wrap over conf->masters[]) + thin `xce_failover_rotate` (close-old/reconnect same endpoint). Live connect path is E2E (6.2). LAYERING: functions taking the private `api_ctx_t` are declared in `src/lib/client_internal.h`, NOT the public `xce/client_failover.h` (which keeps only the pure int-typed classifier) — per src/lib/CLAUDE.md (public headers in src/include/xce/ must not pull private state).
- 2.3: RedirectCapture via non-static seam `xce_ctx_capture_redirect_hint()` called in handle_reply BEFORE the capnp free AND before the `status<0` early return (so negative E_XCE_REDIRECT still captures). Bounded strncpy into redirect_hint_host. ReplyWaitWake needed NO production change — existing `handle_disconnect` (err_handler, xcectx.c) already injects -ECONNRESET into wbuf + sets XCE_CTX_ERR; full err_handler path is covered by E2E (6.2).
- 3.1: pure `xcecli_retry_decide` (no I/O — classify+budget+jittered-backoff calc, D6 exhaustion: no-primary→-E_XCE_NOT_READY, transport→keep last rc) + thin `xcecli_retry_next` (decide + RETRY_BACKOFF nanosleep + xce_failover_rotate + tlog_info/tlog_warn). Logger = utils.h tlog_* macros. State-reset/drain deferred to 3.2; rejoin hook to 4.1.
- 3.2: `xcecli_ctx_recover` (socket-free): flips ac->ctx.state ERR→RUN ONLY, drains stale wbuf via non-blocking cbuf_tryget (-EAGAIN on empty); wired into xcecli_retry_next after successful rotate, before return true. THIS is the -EINVAL bug fix. Added missing prototype `extern int cbuf_tryget(cbuf_t*,void*)` to mylock.h (function already defined mylock.c:166, just undeclared).
- 4.1: public `xcecli_reconnect_cb` typedef + `xcecli_set_reconnect_handler` (client.h/client.c, stores cb/arg in failover); socket-free `xcecli_invoke_rejoin` wired into xcecli_retry_next AFTER rotate+recover, BEFORE resume. hook<0 = failed attempt → bounded tail-recursion (consumes attempt budget, capped at max_retries). typedef MOVED from client_internal.h to client.h. API additive.
- 5.1: `XCECLI_FAILOVER_CALL(ac,is_write,EXPR)` macro (client_internal.h) = lock(failover.lock) → retry init → do{rc=EXPR}while(xcecli_retry_next) → unlock → yield rc. Wrapped 18 reply-bearing ops; writes=true for jobsubmit/jobdispatch/jobctrl/queuectrl/hostctrl/update_resvals/update_jusage/runtask/join/set_pend_reason/clear_reason, reads=false otherwise. R1.3: zero signature changes. DESIGN CORRECTION: `xce_nop` is fire-and-forget (no cbuf_get, xceapi.c:30) — moved nop reply-read→fire-and-forget in design OpCategory; nop wraps in 5.2 not 5.1.
- 5.2: `XCECLI_FF_CALL` macro (single-shot, NO retry loop/backoff/budget): lock → if known-dead (xcecli_ff_needs_reconnect = ctx.state==XCE_CTX_ERR) one rotate+recover → one send → unlock. Applied at the 3 wire chokepoints xcecli_notify/heartbeat/nop (notify_* delegate to xcecli_notify). wakeup/try_wakeup are LOCAL-only (cbuf_tryput own wbuf, no wire send) so correctly NOT wrapped.
- 5.3: `fill_redirect_reply(xce_reply_t*, const ha_state_t*)` (non-static, join.c) sets redirect_host=ha.primary_host + redirect_port=(uint16_t)ha.primary_port; called ONLY on the E_XCE_REDIRECT branch before xce_reply. Audit: join.c is the SOLE E_XCE_REDIRECT emitter. NOMASTER path unchanged (8.2). host alias (no copy) safe: ha lives cluster-lifetime, xce_reply serializes synchronously. New test_join_redirect.c registered in controller_test.c + meson.build.
- 6.1: full lib suite green (161 tests). All 14 reqs (2.4,2.5,3.2,3.3,4.1-4.5,5.2,6.2,6.3,8.3,8.4) map to genuine passing tests. Gap-filled 6.3 with `test_decide_jitter_staggers_many_clients` (64 distinct-seed carriers → staggered delays, anti-thundering-herd). No production bugs found.
- 6.2 (LIVE E2E — partial): Live 3-node cluster (deploy-ha.sh) was brought up and DID elect a primary; killing the primary triggered server-side failover (ctrl-2 self-elected term=2). The run REPRODUCED the original `Invalid argument` client bug → root-caused + FIXED (bug-609, known-dead-ctx override) + regression unit test added. The clean within-session client transparent-recovery could NOT be visually demonstrated in this sandbox (background controller processes are reaped between tool calls; nix runtime env needed for backgrounded procs; lib build cleans the controller binary). RECOMMEND final visual confirmation in a stable env: build all, `/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993`, run xce_test (set masters / client / jobsub / joblist), kill the primary, confirm joblist reports cluster-not-ready then transparently recovers (no EINVAL).
- 6.2 (LIVE, round 2 — your env): failover now ENGAGES (xcecli_joblist→xcecli_retry_next→xce_failover_rotate). Two live-only bugs found+fixed: bug-609 (fresh-call-after-disconnect -EINVAL classified terminal → known-dead-ctx override in decide) and bug-610 (xce_failover_rotate SEGV in endpoint_connect — reconnect never re-bound ep->iomgr nor re-armed owner/msg/err handlers + iomgr_monitor; fixed via new `xce_ctx_reconnect()` in xcectx.c that mirrors xcecli_start establishment without restarting iomgr/vtaskmgr or touching ctx->state). REBUILD controller before retest: `cd src && nix develop --command bash tools/xce-build.sh controller` (relinks xce_test). NOTE: building `lib` removes the controller binary from the shared build dir — always rebuild controller before deploy.