2.5 KiB
| id | status | status_updated | status_note | component | severity | timestamp | files | tags | related | fix_commits | legacy | gitea_issue | gitea_synced_status | gitea_synced_hash | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1850 | fixed | 2026-09-25 | 6b1862f8; soaks 2026-09-25: 0 NOT_READY-triggered rotations, 1 live same-controller retry (worker, recovered in 1.2s), all PASS |
lib | unknown | 2026-09-24T05:30:00Z |
|
|
|
true | 274 | fixed | f647538f9468 |
bug-1850: root trigger of bug-1846 part 2's mid-round rejoins: the live scheduler reconnected/re-joined…
Description
OPEN, NOT FIXED -- root trigger of bug-1846 part 2's mid-round rejoins: the live scheduler reconnected/re-joined roughly every ~100s in scheduler-replay-20260923-085356.replay (each rejoin = full config/worker/job re-push from on_scheduler_join()).
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
UNCONFIRMED (code reading only; the live scheduler.log for the capture window isn't available here). The controller never closes a scheduler connection proactively (only handle_disconnect() on a socket error sets XCE_CTX_ERR). But xcecli_retry_next() calls xce_failover_rotate() -> xce_ctx_reconnect() on EVERY retry, and xce_failover_classify() treats ETIMEDOUT (any request whose reply takes >10s, cbuf_deadline_10s in xceapi.c), E_XCE_NOT_READY and E_XCE_REDIRECT/STANDBY as retryable. So a slow or 'not ready' reply to a dispatch/pend-reason publish from a HEALTHY primary rotates the scheduler to the next master (typically a standby, which redirects back), and each reconnect fires the rejoin callback -> sched_check_pending_rejoin() -> xcecli_join() -> full re-push, which adds load and can self-reinforce. Also: a reply arriving after its 10s deadline lands in the capacity-1 wbuf and can be consumed by the NEXT request (xcecli_ctx_recover() drains only one stale entry).
Fix
NOT FIXED. Next step: grep the live scheduler.log for 'publish retry-budget exhausted', 'xcecli failover', 'pending rejoin succeeded' around each truncated round to identify the triggering rc. Any fix changes libxce HA client failover semantics (e.g. not rotating away from a primary on ETIMEDOUT/NOT_READY) -> spec workflow per CLAUDE.md, not a drive-by.
Verification
Not recorded (migrated from buglog.json).
References
- Gitea: #274