lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(xceadmin): accept a comma-separated --controller failover list (bug-1821, bug-1822)

Browse source 
xceadmin's --controller option (config push, and scheduler record
start/stop/status) accepted only a single host:port. Passing every
configured controller as a comma-separated list -- the natural way to
query an HA cluster without knowing which node is alive/primary --
mis-parsed the whole joined string as one bogus hostname and failed
with a DNS-resolution error.

Adds xceadmin.discovery.resolve_controllers() (comma-split candidate
parsing) and follow_redirects_any() (tries each candidate in order,
skipping ones that are unreachable at the network level -- connection
refused, DNS failure, or a discovery timeout -- mirroring xcecli/
xce_test's own client-side master-rotation failover). A candidate that
IS reachable but returns a genuine protocol-level failure is not
retried against a different node. push.py and scheduler.py's own
duplicated single-host _split_controller helpers are removed in favor
of this shared implementation.

Also fixes a related gap bug-1821's own live verification surfaced:
right after a primary is killed, a standby doesn't start a new
election until its own heartbeat timeout elapses (~10s), so it keeps
redirecting every candidate to the now-dead old primary -- exhausting
the whole candidate list and raising NoPrimaryError, which
_send_admin_request()'s existing transient-retry loop (bug-1713/1715)
never caught. That loop now retries on NoPrimaryError too, using its
own (longer) budget sized for the heartbeat-timeout-plus-election
window rather than the RPC-level "already electing" budget the
original retry logic was tuned for.

TDD RED->GREEN, 304/304 tests passing. Live-verified end to end: a
comma-list correctly skips a dead first candidate and reaches a live
one on a healthy cluster, and correctly rides out a real primary kill
and re-election (retry loop visibly cycling ~15s) to reach the newly-
elected primary rather than failing immediately.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-17 03:53:35 +00:00
parent baa07d78db
commit 723fdb637b
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
8 changed files with 521 additions and 82 deletions
Download patch file Download diff file Expand all files Collapse all files

10
src/xceadmin/tests/test_config_push_mock.py
View file

@ -24,7 +24,7 @@ import pytest
from xcecommon import wire
from xcecommon.errors import ConnectionTimeoutError, NoPrimaryError, ReplyDecodeError, TooManyRedirectsError
from xcecommon.schema import load_schema
from xceadmin import push
from xceadmin import discovery, push
def _server_context(cert_file: Path, key_file: Path) -> ssl.SSLContext:
@ -228,13 +228,17 @@ def test_push_config_rejects_bare_requestreply_bytes_old_wire_shape_regression(
def test_push_config_default_port_used_when_omitted():
host, port = push._split_controller("somehost")
# push.py delegates --controller parsing to xceadmin.discovery.
# resolve_controllers (shared with scheduler.py) -- see test_discovery.py
# for that function's own dedicated coverage; this test just confirms
# push_config's pipeline still reaches it for the single-host case.
(host, port), = discovery.resolve_controllers("somehost")
assert host == "somehost"
assert port == 9991
def test_push_config_explicit_port_parsed():
host, port = push._split_controller("somehost:1234")
(host, port), = discovery.resolve_controllers("somehost:1234")
assert host == "somehost"
assert port == 1234