lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(xceadmin): accept a comma-separated --controller failover list (bug-1821, bug-1822)

Browse source 
xceadmin's --controller option (config push, and scheduler record
start/stop/status) accepted only a single host:port. Passing every
configured controller as a comma-separated list -- the natural way to
query an HA cluster without knowing which node is alive/primary --
mis-parsed the whole joined string as one bogus hostname and failed
with a DNS-resolution error.

Adds xceadmin.discovery.resolve_controllers() (comma-split candidate
parsing) and follow_redirects_any() (tries each candidate in order,
skipping ones that are unreachable at the network level -- connection
refused, DNS failure, or a discovery timeout -- mirroring xcecli/
xce_test's own client-side master-rotation failover). A candidate that
IS reachable but returns a genuine protocol-level failure is not
retried against a different node. push.py and scheduler.py's own
duplicated single-host _split_controller helpers are removed in favor
of this shared implementation.

Also fixes a related gap bug-1821's own live verification surfaced:
right after a primary is killed, a standby doesn't start a new
election until its own heartbeat timeout elapses (~10s), so it keeps
redirecting every candidate to the now-dead old primary -- exhausting
the whole candidate list and raising NoPrimaryError, which
_send_admin_request()'s existing transient-retry loop (bug-1713/1715)
never caught. That loop now retries on NoPrimaryError too, using its
own (longer) budget sized for the heartbeat-timeout-plus-election
window rather than the RPC-level "already electing" budget the
original retry logic was tuned for.

TDD RED->GREEN, 304/304 tests passing. Live-verified end to end: a
comma-list correctly skips a dead first candidate and reaches a live
one on a healthy cluster, and correctly rides out a real primary kill
and re-election (retry loop visibly cycling ~15s) to reach the newly-
elected primary rather than failing immediately.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-17 03:53:35 +00:00
parent baa07d78db
commit 723fdb637b
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
8 changed files with 521 additions and 82 deletions
Download patch file Download diff file Expand all files Collapse all files

57
.wolf/buglog.json
View file

@ -22317,6 +22317,63 @@
],
"occurrences": 1,
"last_seen": "2026-09-17T02:45:00Z"
},
{
"id": "bug-1821",
"timestamp": "2026-09-17T04:10:00Z",
"error_message": "User-reported: xceadmin's --controller option (config push and all 3 scheduler record start/stop/status commands) accepted only a SINGLE host:port -- passing a comma-separated list of all configured controllers (the natural way to query an HA cluster without knowing in advance which node is alive/primary, e.g. `--controller \"localhost:9901,localhost:9902,localhost:9903\"` with 9901 dead) failed outright: `Error: cannot connect to localhost:9901,localhost:9902,localhost:9903: [Errno -2] Name or service not known` -- the whole comma-joined string was parsed as one bogus hostname.",
"file": "src/xceadmin/src/xceadmin/discovery.py (new resolve_controllers/follow_redirects_any), src/xceadmin/src/xceadmin/push.py, src/xceadmin/src/xceadmin/scheduler.py (both migrated off their own duplicated _split_controller), src/xceadmin/src/xceadmin/cli.py (--controller help text)",
"root_cause": "push.py and scheduler.py each had their own private _split_controller(controller: str) -> (host, port) that did a single controller.rsplit(':', 1) -- designed only ever to see one host:port. A comma-separated value was never split on ',' at all; it fell through as one literal (unparseable) hostname straight into socket.create_connection via xcecommon.wire.connect, which propagates the resulting socket.gaierror (a plain OSError subclass) unwrapped by design.",
"fix": "FIXED (2026-09-17, TDD RED->GREEN). Added xceadmin.discovery.resolve_controllers(controller) -> list[(host,port)] (comma-split, whitespace-stripped, per-entry bare-host default port) and xceadmin.discovery.follow_redirects_any(candidates, send_fn) -> (host,port,reply_bytes), which tries each candidate in order via the existing follow_redirects(), catching per-candidate OSError (connection refused, DNS failure) and NoPrimaryError (discovery-level timeout) and moving to the next candidate -- but NOT catching ReplyDecodeError/TooManyRedirectsError, since a genuinely-reachable candidate's protocol-level failure isn't fixed by trying a different node. push.py/scheduler.py's own _split_controller removed; both now call resolve_controllers + follow_redirects_any. cli.py's --controller help text updated on all 4 commands to document the comma-list form. 12 new unit tests in test_discovery.py (single/list/whitespace parsing; first-reachable-wins; skip-refused/skip-DNS-failure/skip-timeout-then-succeed; all-fail raises NoPrimaryError naming every candidate; redirect-following composes correctly with failover; a reachable candidate's decode error does NOT fall through to the next). 2 existing tests per file (test_config_push_mock.py, test_scheduler.py) updated to call the new shared function instead of the removed private one. Full suite 304/304. Live-verified: deployed a 3-node cluster, killed the first-listed controller, confirmed `--controller \"dead,live1,live2\"` correctly skips the dead one and reaches a live one (both healthy-cluster and killed-primary scenarios).",
"tags": [
"xceadmin",
"discovery.py",
"push.py",
"scheduler.py",
"cli.py",
"--controller",
"failover",
"master-rotation",
"user-reported",
"fixed",
"tdd",
"live-verified"
],
"related_bugs": [
"bug-1566",
"bug-1822"
],
"occurrences": 1,
"last_seen": "2026-09-17T04:10:00Z"
},
{
"id": "bug-1822",
"timestamp": "2026-09-17T04:10:00Z",
"error_message": "Found live while verifying bug-1821's --controller failover-list fix: querying `xceadmin scheduler record status --controller \"host1,host2,host3\"` immediately after killing the primary hard-failed with `Error: no primary available: none of the configured controllers responded (...)` instead of riding out the cluster's own re-election, even though _send_admin_request() already has a bounded retry loop (bug-1713/1715) built for exactly this 'cluster still settling' condition.",
"file": "src/xceadmin/src/xceadmin/scheduler.py (_send_admin_request)",
"root_cause": "Right after a primary is killed, a standby does not even START a new election until its own heartbeat_timeout_ms elapses (ha.h: HA_DEFAULT_HEARTBEAT_TIMEOUT_MS = 10000, i.e. 10s) -- until then it keeps redirecting every schedAdmin request straight back to the now-dead old primary. follow_redirects_any() (bug-1821) correctly detects that dead redirect target is unreachable and tries the next configured candidate, but if EVERY candidate is in this same stale-redirect state, the whole candidate list is exhausted and follow_redirects_any raises NoPrimaryError -- an exception _send_admin_request()'s existing retry loop never caught. That loop only retried on a REPLY that arrived and said not-ready (RequestReply.status in _TRANSIENT_STATUSES, or SchedAdminReply.ok=false with a known-transient reason) -- a total connection failure (no reply at all) propagated immediately with zero retries, even though it is the identical 'cluster still settling' condition the existing retry logic was built to ride out, just signaled differently.",
"fix": "FIXED (2026-09-17, TDD RED->GREEN). _send_admin_request()'s retry loop restructured from a single bounded for-loop sharing one attempt counter into a `while True` loop with TWO independent counters/budgets: the pre-existing status_attempt (_MAX_TRANSIENT_ATTEMPTS=4, _TRANSIENT_RETRY_DELAY_S=1.5s, unchanged, still shared by the reply.status and SchedAdminReply.ok=false branches exactly as before) and a new no_primary_attempt (_MAX_NO_PRIMARY_ATTEMPTS=8, _NO_PRIMARY_RETRY_DELAY_S=2.0s -- deliberately larger, since this failure mode's worst case is the ~10s heartbeat-timeout detection window plus real election time, categorically longer than the RPC-level 'already electing, replies not-ready' scenario the original budget was tuned for) that catches NoPrimaryError from follow_redirects_any and retries the whole candidate list fresh. 2 new unit tests (monkeypatching follow_redirects_any directly: retries-then-succeeds, and exhaustion-still-propagates-NoPrimaryError after exactly _MAX_NO_PRIMARY_ATTEMPTS attempts). Full suite 304/304. Live-verified: killed a settled cluster's primary, immediately queried the failover-list, confirmed (in one repro) the retry loop visibly cycled the candidate list 5+ times over ~15s while standbys still redirected to the dead primary, then successfully reached the genuinely newly-elected primary once the standbys' own election completed -- the query result correctly transitioned from NoPrimaryError to the new primary's real (if scheduler-daemon-absent-in-this-test-env) reply, proving the retry rode out the real HA re-election, not just a mocked delay.",
"tags": [
"xceadmin",
"scheduler.py",
"_send_admin_request",
"NoPrimaryError",
"retry",
"heartbeat_timeout_ms",
"election",
"live-verified",
"fixed",
"tdd",
"bug-1713",
"bug-1715"
],
"related_bugs": [
"bug-1713",
"bug-1715",
"bug-1821"
],
"occurrences": 1,
"last_seen": "2026-09-17T04:10:00Z"
}
]
}