Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
3.9 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1822 | fixed | 2026-09-24 | xceadmin | unknown | 2026-09-17T04:10:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1822: Found live while verifying bug-1821's --controller failover-list fix: querying `xceadmin scheduler…
Description
Found live while verifying bug-1821's --controller failover-list fix: querying xceadmin scheduler record status --controller "host1,host2,host3" immediately after killing the primary hard-failed with Error: no primary available: none of the configured controllers responded (...) instead of riding out the cluster's own re-election, even though _send_admin_request() already has a bounded retry loop (bug-1713/1715) built for exactly this 'cluster still settling' condition.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
Right after a primary is killed, a standby does not even START a new election until its own heartbeat_timeout_ms elapses (ha.h: HA_DEFAULT_HEARTBEAT_TIMEOUT_MS = 10000, i.e. 10s) -- until then it keeps redirecting every schedAdmin request straight back to the now-dead old primary. follow_redirects_any() (bug-1821) correctly detects that dead redirect target is unreachable and tries the next configured candidate, but if EVERY candidate is in this same stale-redirect state, the whole candidate list is exhausted and follow_redirects_any raises NoPrimaryError -- an exception _send_admin_request()'s existing retry loop never caught. That loop only retried on a REPLY that arrived and said not-ready (RequestReply.status in _TRANSIENT_STATUSES, or SchedAdminReply.ok=false with a known-transient reason) -- a total connection failure (no reply at all) propagated immediately with zero retries, even though it is the identical 'cluster still settling' condition the existing retry logic was built to ride out, just signaled differently.
Fix
FIXED (2026-09-17, TDD RED->GREEN). _send_admin_request()'s retry loop restructured from a single bounded for-loop sharing one attempt counter into a while True loop with TWO independent counters/budgets: the pre-existing status_attempt (_MAX_TRANSIENT_ATTEMPTS=4, _TRANSIENT_RETRY_DELAY_S=1.5s, unchanged, still shared by the reply.status and SchedAdminReply.ok=false branches exactly as before) and a new no_primary_attempt (_MAX_NO_PRIMARY_ATTEMPTS=8, _NO_PRIMARY_RETRY_DELAY_S=2.0s -- deliberately larger, since this failure mode's worst case is the ~10s heartbeat-timeout detection window plus real election time, categorically longer than the RPC-level 'already electing, replies not-ready' scenario the original budget was tuned for) that catches NoPrimaryError from follow_redirects_any and retries the whole candidate list fresh. 2 new unit tests (monkeypatching follow_redirects_any directly: retries-then-succeeds, and exhaustion-still-propagates-NoPrimaryError after exactly _MAX_NO_PRIMARY_ATTEMPTS attempts). Full suite 304/304. Live-verified: killed a settled cluster's primary, immediately queried the failover-list, confirmed (in one repro) the retry loop visibly cycled the candidate list 5+ times over ~15s while standbys still redirected to the dead primary, then successfully reached the genuinely newly-elected primary once the standbys' own election completed -- the query result correctly transitioned from NoPrimaryError to the new primary's real (if scheduler-daemon-absent-in-this-test-env) reply, proving the retry rode out the real HA re-election, not just a mocked delay.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1713, bug-1715, bug-1821