Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
8.6 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1713 | fixed | 2026-09-24 | xceadmin | unknown | 2026-08-26T17:15:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1713: User-reported, live 3-node cluster, right after bug-1700/bug-1701's redirect fix landed and was…
Description
User-reported, live 3-node cluster, right after bug-1700/bug-1701's redirect fix landed and was live-tested: xceadmin scheduler record status --controller <standby> now correctly redirects ('redirected: localhost:9903 -> localhost:9901'), but the SAME command against the SAME redirect target intermittently fails with Error: controller returned status -10013 (sometimes succeeds with real recording status, sometimes fails on an immediate repeat) instead of always succeeding.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
FULLY ROOT-CAUSED. -10013 is E_XCE_NOT_READY (errnox.h: XCE_ERRNO_BASE+12, 'Controller in CTRL_INIT or CTRL_ELECTING state'). The redirect from bug-1701's fix correctly points at whichever node was primary as of its last ctrlhello, but that node can be transiently mid-(re-)election by the time the redirected request actually arrives -- expected/normal on a freshly-booted cluster (election churn during the first several seconds), or any time a real failover is in flight. ha_dispatch_gate() returns -XCE_ERR_NOT_READY for exactly this transient window, with NO redirect populated (by design -- a CTRL_INIT/CTRL_ELECTING node genuinely has no settled primary to point to yet). The sibling code E_XCE_NOMASTER (errnox.h:22) carries an explicit design-intent comment: 'Standby has no known primary -- client should retry' -- but _send_admin_request() (scheduler.py:119-161) treats EVERY nonzero reply.status, including both E_XCE_NOT_READY and E_XCE_NOMASTER, as an immediate hard SchedAdminError with no retry at all. This is a documented client-side gap, not a server bug: the server-side behavior (refuse with a specific, distinguishable transient-state code) is correct and intentional; the client simply never implements the retry the errno's own comment already promises.
Fix
FIXED and LIVE-VERIFIED (2026-08-26). _send_admin_request() (src/xceadmin/src/xceadmin/scheduler.py:119-231) now wraps its whole request/redirect/decode cycle in a bounded retry loop, re-running follow_redirects() FROM SCRATCH each attempt (not a bare resend to the same host:port) -- a fresh attempt can discover a newly-settled or different primary once an election concludes. Retries EXACTLY reply.status in (_E_XCE_NOMASTER, _E_XCE_NOT_READY) (module-level constants at scheduler.py:80-111, values -10011/-10013 -- NEGATIVE, matching the real wire value confirmed via join.c:284 rc = -E_XCE_NOMASTER / commit.c:259 and controller.capnp:608's signed Int32 status field). E_XCE_STANDBY (-10014) is deliberately excluded -- untouched, still resolved purely by follow_redirects()'s existing redirect-following (bug-1701); a standby reply that DOES carry a redirect never even reaches this status check (follow_redirects consumes it internally), and one that does NOT carry a redirect is a genuine non-transient rejection that must keep failing immediately. No shared Python-side mirror of errnox.h's enum exists anywhere in xcecommon/xceadmin (confirmed by grepping the whole package) -- followed the one existing precedent instead of inventing a new convention: test_e2e_live_ha_push.py's own local _E_XCE_STANDBY = 10014 constant with an errnox.h-citing comment. Retry budget: _MAX_TRANSIENT_ATTEMPTS=4 total attempts (1 initial + 3 retries), _TRANSIENT_RETRY_DELAY_S=1.5 fixed delay between attempts (time.sleep) -- sized at the same scale as this project's existing bounded-retry shapes (follow_redirects' own _MAX_REDIRECT_HOPS=3; the HA soak driver's verify_classify retry loop) since election-churn settling is single-digit-seconds, not a long outage; no exponential backoff needed at this scale. Retry-budget exhaustion still raises the existing SchedAdminError (no new exception type needed) but with a message appending '(still transient after N attempts over ~Xs -- giving up)' for clear distinguishability from an immediate non-retried hard failure. RED/GREEN proof (src/xceadmin/tests/test_scheduler.py): added _SequentialMockController (a finite, exact-connection-count multi-TLS-connection mock server, since the retry loop reconnects from scratch each attempt -- distinct from the file's existing one-shot _MockController) plus 5 new tests -- test_record_status_retries_on_transient_status_then_succeeds (parametrized NOT_READY/NOMASTER, 2 failures then success -> eventual success, FAILED pre-fix at collection time since the referenced constants didn't exist yet, and confirmed behaviorally via an ad hoc script that the pre-fix code raises immediately on a single transient reply with only 1 connection attempt), test_record_status_retry_exhaustion_raises_distinguishable_error (always-transient mock, scheduler.time.sleep monkeypatched so the test does not actually wait; asserts exact attempt count + distinguishable message), test_record_status_standby_without_redirect_not_retried and test_record_status_standby_with_redirect_follows_redirect_not_retried (regression locks: E_XCE_STANDBY is never retried, with/without a populated redirect, zero sleep calls in either case) -- all 5 PASS post-fix in 0.20s total (no real sleeping). Full xceadmin suite: 274 passed, 1 failed -- the failure (test_live_config_push_against_standby_returns_standby_rejection_not_redirect, test_e2e_live_ha_push.py) is PRE-EXISTING and unrelated: confirmed via git stash push -- <only scheduler.py/test_scheduler.py> that it reproduces IDENTICALLY on the unmodified codebase -- it is a live e2e test whose own docstring already anticipated going stale once bug-1700/1701's generic dispatch-gate redirect fix (already landed this session) generalizes to OPCode_upgradeConfig, which it now has; zero new failures caused by this fix. LIVE VERIFICATION (real 3-node cluster, deploy-ha.sh + xce.build's real controller binary, editable-install xceadmin venv pointed at scheduler.py so before/after needed no rebuild, only a file swap): a plain fresh-boot tight loop immediately after start-all.sh did NOT naturally reproduce -10013 in this sandbox (election settles with highly variable timing, sometimes near-instantly, since ha_has_write_quorum()'s standby_endpoints are populated synchronously inside raft_cb_become_leader and are typically already satisfied by the time start-all.sh's 1s-granularity health-check first observes CTRL_PRIMARY) -- but a REAL FAILOVER (kill -9 the live primary, then hammer the two survivors with zero sleep, no health-check wait) reliably reproduced the exact user-reported symptom, which this bug's own root-cause writeup already named as an equally valid trigger ('...or any time a real failover is in flight'): pre-fix, the client surfaced a bare 'Error: controller returned status -10013' to the user at t+8.60s post-kill (exit 1); post-fix, an identical failover on a fresh cluster produced ZERO -10013/-10011 failures across the equivalent ~15s post-kill window (52 requests; every non-success was either a legitimate pre-election-completion 'connection refused' to the just-killed primary's now-dead listener, or a legitimate post-gate 'scheduler not connected' business-level SchedAdminError from this sandbox having no real xce_scheduler process deployed -- neither is a transient status code, and correctly reaches the caller unretried in both runs, proving the fix does not overreach). push.py's push_config()/probe_config() (src/xceadmin/src/xceadmin/push.py:92-179) have the IDENTICAL gap -- same two transient codes, same immediate-return-as-data with zero retry -- confirmed by direct code read; NOT fixed here (out of this bug's Python-scheduler-admin-path scope, and push.py's error-vs-return convention differs from scheduler.py's raise-based one, so a proper fix there is more than a trivial shared-helper change) -- flagged as a sibling finding for a future bug, not silently left undocumented.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1700, bug-1701