40 lines
2.5 KiB
Markdown
40 lines
2.5 KiB
Markdown
---
|
|
id: "bug-1850"
|
|
status: "fixed"
|
|
status_updated: "2026-09-25"
|
|
status_note: "6b1862f8; soaks 2026-09-25: 0 NOT_READY-triggered rotations, 1 live same-controller retry (worker, recovered in 1.2s), all PASS"
|
|
component: "lib"
|
|
severity: "unknown"
|
|
timestamp: "2026-09-24T05:30:00Z"
|
|
files: ["src/lib/client_failover.c (xcecli_retry_next / xce_failover_classify)", "src/lib/xceapi.c (10s reply deadline)"]
|
|
tags: ["libxce", "client", "failover", "reconnect", "ha", "scheduler"]
|
|
related: []
|
|
fix_commits: ["6b1862f8"]
|
|
legacy: true
|
|
gitea_issue: 274
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "f647538f9468"
|
|
---
|
|
|
|
# bug-1850: root trigger of bug-1846 part 2's mid-round rejoins: the live scheduler reconnected/re-joined…
|
|
|
|
## Description
|
|
OPEN, NOT FIXED -- root trigger of bug-1846 part 2's mid-round rejoins: the live scheduler reconnected/re-joined roughly every ~100s in scheduler-replay-20260923-085356.replay (each rejoin = full config/worker/job re-push from on_scheduler_join()).
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Root cause
|
|
UNCONFIRMED (code reading only; the live scheduler.log for the capture window isn't available here). The controller never closes a scheduler connection proactively (only handle_disconnect() on a socket error sets XCE_CTX_ERR). But xcecli_retry_next() calls xce_failover_rotate() -> xce_ctx_reconnect() on EVERY retry, and xce_failover_classify() treats ETIMEDOUT (any request whose reply takes >10s, cbuf_deadline_10s in xceapi.c), E_XCE_NOT_READY and E_XCE_REDIRECT/STANDBY as retryable. So a slow or 'not ready' reply to a dispatch/pend-reason publish from a HEALTHY primary rotates the scheduler to the next master (typically a standby, which redirects back), and each reconnect fires the rejoin callback -> sched_check_pending_rejoin() -> xcecli_join() -> full re-push, which adds load and can self-reinforce. Also: a reply arriving after its 10s deadline lands in the capacity-1 wbuf and can be consumed by the NEXT request (xcecli_ctx_recover() drains only one stale entry).
|
|
|
|
## Fix
|
|
NOT FIXED. Next step: grep the live scheduler.log for 'publish retry-budget exhausted', 'xcecli failover', 'pending rejoin succeeded' around each truncated round to identify the triggering rc. Any fix changes libxce HA client failover semantics (e.g. not rotating away from a primary on ETIMEDOUT/NOT_READY) -> spec workflow per CLAUDE.md, not a drive-by.
|
|
|
|
## Verification
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## References
|
|
- Gitea: #274
|