lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(lib): retry a NOT_READY controller in place instead of failing over (bug-1850)

Browse source 
xce_failover_classify() mapped E_XCE_NOT_READY with NOMASTER to
RETRY_BACKOFF, so xcecli_retry_next() rotated to the next master,
reconnected and fired the rejoin hook. But a NOT_READY reply comes from a
controller that answered on a working connection -- here a healthy primary
that had briefly lost its write quorum (both standbys NACKing, bug-1864).
The scheduler abandoned it for the standbys, rejoined, and got a full
re-push built from lagging committed state (21:39:24-27 in data/logs).

New action XCE_FO_RETRY_SAME (appended; existing values unchanged):
NOT_READY backs off (same jittered schedule, same -E_XCE_NOT_READY on
budget exhaustion) and re-sends on the current connection -- no rotate,
reconnect or rejoin. NOMASTER keeps rotating after backoff. If the
controller is really a standby it answers REDIRECT/STANDBY/NOMASTER next
time, and if it dies the transport error rotates, as before.

Tests (test_libxce_retry.c): taxonomy row NOT_READY -> RETRY_SAME;
decide backs off and exhausts to -E_XCE_NOT_READY; the real
xcecli_retry_next keeps cur_master on NOT_READY (rotated before) while
STANDBY still rotates; backoff-growth test moved to NOMASTER. Failover
tests 56/56; lib, controller 5/5, worker 10/10, scheduler 87/87,
harness 5/5.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-25 15:04:47 +00:00
parent 910b43cd37
commit 6b1862f8d3
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
4 changed files with 100 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

30
src/lib/client_failover.c
View file

@ -59,11 +59,19 @@ xce_fo_action_t xce_failover_classify(int status, bool is_write, int attempt)
if (code == E_XCE_REDIRECT || code == E_XCE_STANDBY)
return XCE_FO_RETRY_NOW;
/* No primary known yet / controller initializing or electing — rotate after
* jittered backoff (4.1). */
if (code == E_XCE_NOMASTER || code == E_XCE_NOT_READY)
/* No primary known yet — rotate after jittered backoff (4.1). */
if (code == E_XCE_NOMASTER)
return XCE_FO_RETRY_BACKOFF;
/* bug-1850: the controller answered, so this connection works; it is only
* briefly not ready (electing, or a primary that lost its write quorum or
* hit commit-wait admission). Back off and retry the SAME controller --
* rotating away from a healthy primary forced a reconnect + rejoin + full
* re-push. If it turns out to be a standby it answers REDIRECT/STANDBY/
* NOMASTER next time, and if it dies the transport error rotates. */
if (code == E_XCE_NOT_READY)
return XCE_FO_RETRY_SAME;
/* Transport-level failures — reconnect/rotate and retry now (4.4). */
if (code == ECONNREFUSED || code == ETIMEDOUT || code == ECONNRESET)
return XCE_FO_RETRY_NOW;
@ -304,6 +312,7 @@ bool xcecli_retry_decide(api_ctx_t* ac, int* rc, bool is_write,
case XCE_FO_RETRY_NOW:
case XCE_FO_RETRY_BACKOFF:
case XCE_FO_RETRY_SAME:
break;
}
@ -313,14 +322,15 @@ bool xcecli_retry_decide(api_ctx_t* ac, int* rc, bool is_write,
/* Budget exhausted. Distinguish the cause (design D6): a no-primary
* window maps to the distinct cluster-not-ready code; a transport-only
* outage keeps the last transport rc so the caller still sees it. */
if (action == XCE_FO_RETRY_BACKOFF)
if ((action == XCE_FO_RETRY_BACKOFF) || (action == XCE_FO_RETRY_SAME))
*rc = -E_XCE_NOT_READY;
return false;
}
/* Compute the pre-attempt delay: none for prompt rotation (RETRY_NOW),
* a jittered exponential backoff for the no-primary case (RETRY_BACKOFF). */
st->delay_ms = (action == XCE_FO_RETRY_BACKOFF)
st->delay_ms = ((action == XCE_FO_RETRY_BACKOFF)
|| (action == XCE_FO_RETRY_SAME))
? fo_backoff_delay_ms(cfg, st->attempt, &st->rand_seed)
: 0;
return true;
@ -361,6 +371,16 @@ bool xcecli_retry_next(api_ctx_t* ac, int* rc, bool is_write,
if (action == XCE_FO_RETRY_BACKOFF)
fo_sleep_ms(st->delay_ms);
/* bug-1850: RETRY_SAME -- back off and re-send on the current connection;
* no rotate, no reconnect, no rejoin. */
if (action == XCE_FO_RETRY_SAME) {
fo_sleep_ms(st->delay_ms);
INFO("xcecli failover: attempt %d retrying the same controller after "
"%dms (status=%d: not ready)",
st->attempt, st->delay_ms, *rc);
return true;
}
/* Re-point the endpoint to the redirect hint or the next master and
* reconnect (2.x, 4.2, 4.4). Log the rotation target (7.1). A rotate
* failure is not fatal here: the next xce_* attempt observes the dead