lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(lib): retry a NOT_READY controller in place instead of failing over (bug-1850)

Browse source 
xce_failover_classify() mapped E_XCE_NOT_READY with NOMASTER to
RETRY_BACKOFF, so xcecli_retry_next() rotated to the next master,
reconnected and fired the rejoin hook. But a NOT_READY reply comes from a
controller that answered on a working connection -- here a healthy primary
that had briefly lost its write quorum (both standbys NACKing, bug-1864).
The scheduler abandoned it for the standbys, rejoined, and got a full
re-push built from lagging committed state (21:39:24-27 in data/logs).

New action XCE_FO_RETRY_SAME (appended; existing values unchanged):
NOT_READY backs off (same jittered schedule, same -E_XCE_NOT_READY on
budget exhaustion) and re-sends on the current connection -- no rotate,
reconnect or rejoin. NOMASTER keeps rotating after backoff. If the
controller is really a standby it answers REDIRECT/STANDBY/NOMASTER next
time, and if it dies the transport error rotates, as before.

Tests (test_libxce_retry.c): taxonomy row NOT_READY -> RETRY_SAME;
decide backs off and exhausts to -E_XCE_NOT_READY; the real
xcecli_retry_next keeps cur_master on NOT_READY (rotated before) while
STANDBY still rotates; backoff-growth test moved to NOMASTER. Failover
tests 56/56; lib, controller 5/5, worker 10/10, scheduler 87/87,
harness 5/5.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-25 15:04:47 +00:00
parent 910b43cd37
commit 6b1862f8d3
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
4 changed files with 100 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

9
src/include/xce/client_failover.h
View file

@ -18,10 +18,13 @@ typedef enum {
(DUP on a re-sent write) */
XCE_FO_RETRY_NOW, /* E_XCE_REDIRECT / E_XCE_STANDBY / transport error
— rotate, no backoff */
XCE_FO_RETRY_BACKOFF, /* E_XCE_NOMASTER / E_XCE_NOT_READY — rotate after
jittered backoff */
XCE_FO_TERMINAL /* AUTH/PERM/INVALID/NOENT and other non-retriable
XCE_FO_RETRY_BACKOFF, /* E_XCE_NOMASTER — rotate after jittered backoff */
XCE_FO_TERMINAL, /* AUTH/PERM/INVALID/NOENT and other non-retriable
— surface to caller */
XCE_FO_RETRY_SAME /* E_XCE_NOT_READY (bug-1850) — the controller
answered but is briefly not ready: jittered
backoff, then retry the SAME controller (no
rotate/reconnect/rejoin) */
} xce_fo_action_t;
/*