---
id: "bug-1822"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "xceadmin"
severity: "unknown"
timestamp: "2026-09-17T04:10:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/xceadmin/src/xceadmin/scheduler.py (_send_admin_request)"]
tags: ["xceadmin", "scheduler.py", "_send_admin_request", "NoPrimaryError", "retry", "heartbeat_timeout_ms", "election", "live-verified", "tdd", "bug-1713", "bug-1715"]
related: ["bug-1713", "bug-1715", "bug-1821"]
legacy: true
---

# bug-1822: Found live while verifying bug-1821's --controller failover-list fix: querying `xceadmin scheduler…

## Description
Found live while verifying bug-1821's --controller failover-list fix: querying `xceadmin scheduler record status --controller "host1,host2,host3"` immediately after killing the primary hard-failed with `Error: no primary available: none of the configured controllers responded (...)` instead of riding out the cluster's own re-election, even though _send_admin_request() already has a bounded retry loop (bug-1713/1715) built for exactly this 'cluster still settling' condition.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
Right after a primary is killed, a standby does not even START a new election until its own heartbeat_timeout_ms elapses (ha.h: HA_DEFAULT_HEARTBEAT_TIMEOUT_MS = 10000, i.e. 10s) -- until then it keeps redirecting every schedAdmin request straight back to the now-dead old primary. follow_redirects_any() (bug-1821) correctly detects that dead redirect target is unreachable and tries the next configured candidate, but if EVERY candidate is in this same stale-redirect state, the whole candidate list is exhausted and follow_redirects_any raises NoPrimaryError -- an exception _send_admin_request()'s existing retry loop never caught. That loop only retried on a REPLY that arrived and said not-ready (RequestReply.status in _TRANSIENT_STATUSES, or SchedAdminReply.ok=false with a known-transient reason) -- a total connection failure (no reply at all) propagated immediately with zero retries, even though it is the identical 'cluster still settling' condition the existing retry logic was built to ride out, just signaled differently.

## Fix
FIXED (2026-09-17, TDD RED->GREEN). _send_admin_request()'s retry loop restructured from a single bounded for-loop sharing one attempt counter into a `while True` loop with TWO independent counters/budgets: the pre-existing status_attempt (_MAX_TRANSIENT_ATTEMPTS=4, _TRANSIENT_RETRY_DELAY_S=1.5s, unchanged, still shared by the reply.status and SchedAdminReply.ok=false branches exactly as before) and a new no_primary_attempt (_MAX_NO_PRIMARY_ATTEMPTS=8, _NO_PRIMARY_RETRY_DELAY_S=2.0s -- deliberately larger, since this failure mode's worst case is the ~10s heartbeat-timeout detection window plus real election time, categorically longer than the RPC-level 'already electing, replies not-ready' scenario the original budget was tuned for) that catches NoPrimaryError from follow_redirects_any and retries the whole candidate list fresh. 2 new unit tests (monkeypatching follow_redirects_any directly: retries-then-succeeds, and exhaustion-still-propagates-NoPrimaryError after exactly _MAX_NO_PRIMARY_ATTEMPTS attempts). Full suite 304/304. Live-verified: killed a settled cluster's primary, immediately queried the failover-list, confirmed (in one repro) the retry loop visibly cycled the candidate list 5+ times over ~15s while standbys still redirected to the dead primary, then successfully reached the genuinely newly-elected primary once the standbys' own election completed -- the query result correctly transitioned from NoPrimaryError to the new primary's real (if scheduler-daemon-absent-in-this-test-env) reply, proving the retry rode out the real HA re-election, not just a mocked delay.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1713, bug-1715, bug-1821