---
id: "bug-1850"
status: "fixed"
status_updated: "2026-09-25"
status_note: "6b1862f8; soaks 2026-09-25: 0 NOT_READY-triggered rotations, 1 live same-controller retry (worker, recovered in 1.2s), all PASS"
component: "lib"
severity: "unknown"
timestamp: "2026-09-24T05:30:00Z"
files: ["src/lib/client_failover.c (xcecli_retry_next / xce_failover_classify)", "src/lib/xceapi.c (10s reply deadline)"]
tags: ["libxce", "client", "failover", "reconnect", "ha", "scheduler"]
related: []
fix_commits: ["6b1862f8"]
legacy: true
gitea_issue: 274
gitea_synced_status: "fixed"
gitea_synced_hash: "f647538f9468"
---

# bug-1850: root trigger of bug-1846 part 2's mid-round rejoins: the live scheduler reconnected/re-joined…

## Description
OPEN, NOT FIXED -- root trigger of bug-1846 part 2's mid-round rejoins: the live scheduler reconnected/re-joined roughly every ~100s in scheduler-replay-20260923-085356.replay (each rejoin = full config/worker/job re-push from on_scheduler_join()).

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
UNCONFIRMED (code reading only; the live scheduler.log for the capture window isn't available here). The controller never closes a scheduler connection proactively (only handle_disconnect() on a socket error sets XCE_CTX_ERR). But xcecli_retry_next() calls xce_failover_rotate() -> xce_ctx_reconnect() on EVERY retry, and xce_failover_classify() treats ETIMEDOUT (any request whose reply takes >10s, cbuf_deadline_10s in xceapi.c), E_XCE_NOT_READY and E_XCE_REDIRECT/STANDBY as retryable. So a slow or 'not ready' reply to a dispatch/pend-reason publish from a HEALTHY primary rotates the scheduler to the next master (typically a standby, which redirects back), and each reconnect fires the rejoin callback -> sched_check_pending_rejoin() -> xcecli_join() -> full re-push, which adds load and can self-reinforce. Also: a reply arriving after its 10s deadline lands in the capacity-1 wbuf and can be consumed by the NEXT request (xcecli_ctx_recover() drains only one stale entry).

## Fix
NOT FIXED. Next step: grep the live scheduler.log for 'publish retry-budget exhausted', 'xcecli failover', 'pending rejoin succeeded' around each truncated round to identify the triggering rc. Any fix changes libxce HA client failover semantics (e.g. not rotating away from a primary on ETIMEDOUT/NOT_READY) -> spec workflow per CLAUDE.md, not a drive-by.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Gitea: #274