xce_failover_classify() mapped E_XCE_NOT_READY with NOMASTER to
RETRY_BACKOFF, so xcecli_retry_next() rotated to the next master,
reconnected and fired the rejoin hook. But a NOT_READY reply comes from a
controller that answered on a working connection -- here a healthy primary
that had briefly lost its write quorum (both standbys NACKing, bug-1864).
The scheduler abandoned it for the standbys, rejoined, and got a full
re-push built from lagging committed state (21:39:24-27 in data/logs).
New action XCE_FO_RETRY_SAME (appended; existing values unchanged):
NOT_READY backs off (same jittered schedule, same -E_XCE_NOT_READY on
budget exhaustion) and re-sends on the current connection -- no rotate,
reconnect or rejoin. NOMASTER keeps rotating after backoff. If the
controller is really a standby it answers REDIRECT/STANDBY/NOMASTER next
time, and if it dies the transport error rotates, as before.
Tests (test_libxce_retry.c): taxonomy row NOT_READY -> RETRY_SAME;
decide backs off and exhausts to -E_XCE_NOT_READY; the real
xcecli_retry_next keeps cur_master on NOT_READY (rotated before) while
STANDBY still rotates; backoff-growth test moved to NOMASTER. Failover
tests 56/56; lib, controller 5/5, worker 10/10, scheduler 87/87,
harness 5/5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xcecli_retry_next's "xcecli failover: attempt N -> host:port" line now
carries "(after status=<rc>)". Without it a log cannot tell a dead
connection from a healthy primary's commit-timeout or admission-backpressure
reply -- the distinction bug-1850 (the live scheduler re-joining every
~100s) needs to find its triggering rc. Diagnostic only; no behavior change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_failover_rotate reconnected with endpoint_fini+endpoint_init+endpoint_connect
but never re-bound ep->iomgr (reset by endpoint_init) nor re-armed the owner/
msg/err handlers + iomgr_monitor. endpoint_connect then dereferenced a NULL
iomgr (->ssl_ctx) → SEGV at netio.c:672 during live master failover. Added
xce_ctx_reconnect() in xcectx.c that mirrors xcecli_start's establishment
(without restarting iomgr/vtaskmgr or touching ctx->state); rotate delegates to
it. Found via live 3-node E2E. bug-610.
Live 3-node E2E revealed the reported 'Invalid argument' bug was still
reachable: after the primary dies, handle_disconnect leaves ctx.state=
XCE_CTX_ERR, so a FRESH xce_* call returns -EINVAL from the xceapi guard
before reaching a primary. The classifier mapped -EINVAL to TERMINAL, so the
loop bailed without rotating/recovering. decide() now detects the known-dead
ctx and treats it as a prompt reconnect cause (RETRY_NOW), surfacing the
distinct cluster-not-ready code on budget exhaustion, never -EINVAL.
Regression test: test_decide_known_dead_ctx_retries_not_terminal.
