diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 8bcde581..efd83c33 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -19457,6 +19457,35 @@ ], "occurrences": 1, "last_seen": "2026-08-26T09:32:18Z" + }, + { + "id": "bug-1713", + "timestamp": "2026-08-26T17:15:00Z", + "error_message": "User-reported, live 3-node cluster, right after bug-1700/bug-1701's redirect fix landed and was live-tested: `xceadmin scheduler record status --controller ` now correctly redirects ('redirected: localhost:9903 -> localhost:9901'), but the SAME command against the SAME redirect target intermittently fails with `Error: controller returned status -10013` (sometimes succeeds with real recording status, sometimes fails on an immediate repeat) instead of always succeeding.", + "file": "src/xceadmin/src/xceadmin/scheduler.py (_send_admin_request), possibly xceadmin/discovery.py (follow_redirects)", + "root_cause": "FULLY ROOT-CAUSED. -10013 is E_XCE_NOT_READY (errnox.h: XCE_ERRNO_BASE+12, 'Controller in CTRL_INIT or CTRL_ELECTING state'). The redirect from bug-1701's fix correctly points at whichever node was primary as of its last ctrlhello, but that node can be transiently mid-(re-)election by the time the redirected request actually arrives -- expected/normal on a freshly-booted cluster (election churn during the first several seconds), or any time a real failover is in flight. ha_dispatch_gate() returns -XCE_ERR_NOT_READY for exactly this transient window, with NO redirect populated (by design -- a CTRL_INIT/CTRL_ELECTING node genuinely has no settled primary to point to yet). The sibling code E_XCE_NOMASTER (errnox.h:22) carries an explicit design-intent comment: 'Standby has no known primary -- client should retry' -- but `_send_admin_request()` (scheduler.py:119-161) treats EVERY nonzero reply.status, including both E_XCE_NOT_READY and E_XCE_NOMASTER, as an immediate hard SchedAdminError with no retry at all. This is a documented client-side gap, not a server bug: the server-side behavior (refuse with a specific, distinguishable transient-state code) is correct and intentional; the client simply never implements the retry the errno's own comment already promises.", + "fix": "PENDING. Add bounded retry-with-backoff in xceadmin's scheduler-admin client path (_send_admin_request, scheduler.py, and verify whether push.py's config-push client has/needs the identical treatment for the same two codes) specifically for E_XCE_NOMASTER (10011) and E_XCE_NOT_READY (10013) -- both represent 'no settled primary right now, try again shortly', NOT a real failure. Do NOT add retry for E_XCE_STANDBY (10014, now redirect-handled by bug-1701) or any other status code. Suggested shape: a small fixed number of retries (e.g. 3-5) with a short fixed or lightly-backed-off delay (e.g. 1-2s), re-running the SAME follow_redirects() call from the top each attempt (a fresh attempt may get redirected differently once the election settles) -- mirror this project's existing bounded-retry conventions (e.g. verify_classify's retry loop in the HA soak driver, or follow_redirects' own _MAX_REDIRECT_HOPS=3 pattern) rather than inventing a new one. Needs a unit test proving: (a) a mocked server returning E_XCE_NOMASTER/E_XCE_NOT_READY N times then succeeding results in eventual success, not an immediate raise; (b) exceeding the retry budget still raises a clear, distinguishable error (not a silent hang); (c) E_XCE_STANDBY is NOT retried this way (still handled purely by redirect). Live-verify against a freshly-booted real cluster (the exact scenario that surfaced this) showing zero intermittent -10013 failures across N repeated invocations immediately after start-all.sh.", + "tags": [ + "xceadmin", + "scheduler-admin", + "E_XCE_NOT_READY", + "E_XCE_NOMASTER", + "retry", + "transient-state", + "primary-discovery", + "bug-1700-adjacent", + "bug-1701-adjacent", + "found-live", + "root-caused", + "fix-scoped", + "unfixed" + ], + "related_bugs": [ + "bug-1700", + "bug-1701" + ], + "occurrences": 1, + "last_seen": "2026-08-26T17:15:00Z" } ] -} +} \ No newline at end of file