Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
11 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1700 | fixed | 2026-09-24 | controller | unknown | 2026-08-26T16:35:00Z | 2026-09-19T09:00:00Z | 2 |
|
|
|
true |
bug-1700: User-reported: xceadmin scheduler record status (and by the same mechanism, `record…
Description
User-reported: xceadmin scheduler record status (and by the same mechanism, record start/record stop) fails with Error: controller returned status -10014 whenever the --controller/default address happens to point at a controller that is currently a STANDBY, not the primary. User's expectation: 'xceadmin/libxce should find the correct primary to talk to it' instead of surfacing the raw rejection.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
FULLY ROOT-CAUSED, gap is server-side only. -10014 is E_XCE_STANDBY (errnox.h, XCE_ERRNO_BASE+13) -- ha_dispatch_gate() (ha.c:4745-4775) correctly rejects OPCode_schedAdmin on a CTRL_STANDBY node (it is absent from that role's opcode allow-list, by design -- schedAdmin is a primary-only forward-to-scheduler operation). The gate's generic caller in myreq_handler (cluster.c:406-414) sends the rejection reply with reply.status = rc; reply.data_which = RequestReply_data_nodata; and NEVER calls fill_redirect_reply() (join.c:27-31, the exact same one-line helper OPCode_join's handler already uses successfully) -- so redirectHost/redirectPort are left at their zero-value defaults on EVERY dispatch-gate rejection, for EVERY opcode, not just schedAdmin (OPCode_upgradeConfig has the identical, already-documented gap -- see test_e2e_live_ha_push.py's module docstring, confirmed 2026-07-30-era). Critically, this is NOT a client-side deficiency: xceadmin's redirect-following client logic is ALREADY fully implemented, generic/opcode-agnostic, and well-tested (xceadmin.discovery.follow_redirects(), with a full test suite covering 1-hop/3-hop/too-many-hops/timeout cases; xceadmin.scheduler._send_admin_request() already calls it for every record_start/stop/status call). _redirect_target() (discovery.py:80-89) correctly requires BOTH redirectHost and redirectPort to be non-empty/non-zero to treat a reply as a redirect -- given the gate-rejection reply supplies neither, the client correctly concludes there is nothing to follow and surfaces the bare rejection status as-is. The client is not 'giving up' -- it is working exactly per its documented contract given the reply it receives; the whole path already works end-to-end against a properly-behaving server (test_cli_scheduler_record_start_follows_redirect_to_primary already exercises the client side against a mock that DOES send a redirect).
Fix
PARTIALLY FIXED (2026-08-26), live verification FAILS -- see bug-1701 for the blocking root cause, do not close this bug until that is resolved. IMPLEMENTED, unit-tested, and CORRECT exactly as scoped: myreq_handler's generic ha_dispatch_gate rejection path (cluster.c ~410-433) now calls fill_redirect_reply(&reply, &c->ha) when, and only when, rc == -XCE_ERR_STANDBY -- confirmed via ha_dispatch_gate's full body (ha.c:4573-4781) that -XCE_ERR_STANDBY (CTRL_STANDBY's default-case rejection, ha.c:4774) and -XCE_ERR_NOT_READY (CTRL_INIT/CTRL_ELECTING, and CTRL_PRIMARY's config-serve-fence / upgrade-materialization-fence / no-write-quorum branches) are the ONLY two nonzero codes the gate can return -- no third code path exists. -XCE_ERR_NOT_READY is deliberately excluded (no settled primary to redirect to in those states). RED/GREEN proof: added test_schedadmin_standby_with_known_primary_redirects (FAILED pre-fix with 'Expected 10.0.1.7 Was ''', PASSED post-fix) and test_schedadmin_standby_no_known_primary_no_redirect (regression companion, passes in both states) to test_sched_admin_dispatch.c, driving the real myreq_handler via myreq_handler_for_test with OPCode_schedAdmin + CTRL_STANDBY role. Full controller_unit_test: 1200 Tests, 2 Failures both times (pre- and post-fix) -- exactly the pre-existing bug-1561 overlayfs pair (test_conf_trim_keeps_only_n_versions, test_conf_get_saved_tag_no_config), zero new failures. Encode-path NULL-safety CONFIRMED (not assumed): ha_state_t.primary_host is a fixed char[XCE_MAXNAME_LEN] buffer embedded in ha_state_t (ha.h:161), never a pointer, so its address is never NULL; encode_RequestReply (controller.capnp.c:13777-13786) gates strictly on s->redirect_host != NULL and falls back to an explicit empty string/0 otherwise; decode_RequestReply (controller.capnp.c:13841) strdup()s the wire text unconditionally (so a client-side redirect_host is always a real, non-NULL, possibly-empty C string, never NULL) -- an unknown primary therefore safely round-trips as redirectHost='' / redirectPort=0, which xceadmin's discovery.py _redirect_target() (host and port both required truthy) already correctly treats as 'no redirect, use this reply as-is'. No NULL-pointer/crash risk anywhere in this path, confirmed by code inspection of every hop, not assumed. LIVE VERIFICATION FAILS, however: deployed a fresh 3-node cluster twice (deploy-ha.sh, controller+worker+scheduler via scripts/start-all.sh) -- once on a binary built from this fix (git stash of cluster.c to produce a clean before-binary, git stash pop to restore it for the after-binary; xce_test ctrlstatus confirmed ctrl-1 elected primary, ctrl-2/ctrl-3 standby, in both deploys). xceadmin scheduler record status --controller localhost:9992 (a standby) against BOTH the pre-fix and the post-fix binary produced the IDENTICAL result: 'Error: controller returned status -10014', exit 1, no 'redirected: ...' line on stderr either time -- i.e. this fix has ZERO observable effect on the user-reported symptom in a real deployment. Root cause of the no-op: ha_state_t.primary_host/primary_port (ha.h:161-162) are NEVER WRITTEN by any production code path on a standby -- confirmed by grepping every .c file in src/ for an assignment site (none exists outside test fixtures, which set it manually) and by ctrlstatus.c's own pre-existing doc comment (line 38/95: 'the vestigial, never-written ha->primary_host') -- so fill_redirect_reply() always sees an empty string / 0 on a real standby, in every state, permanently, not just during the 'hasn't learned yet' transient window this bug's own original root-cause writeup anticipated and asked to be verified. This is independently corroborated by the ALREADY-CLOSED bug-854 (this bug's own related_bugs entry), whose root_cause section states the identical fact ('primary_host is vestigial/never-written') and whose fix explicitly says 'Deliberately NOT adding server-side primaryName source -- it is only a latency optimization; the client rotation fix alone resolves the hang' for the DIFFERENT ctrlstatus/client-rotation code path. New implication surfaced by this session (see bug-1701): since fill_redirect_reply() is fed by the same permanently-empty field for EVERY caller, OPCode_join's pre-existing, supposedly-successful use of the exact same helper (join.c:277) is, by the same logic, equally non-functional against a live standby in production today -- this was not previously flagged anywhere in the bug log. Per this bug's own instructions, NOT papering over this with a workaround guessed in this session -- stopping for review. The gate-rejection code-path change above is correct, tested, and worth keeping (it is necessary infrastructure and fixes the OPCode_upgradeConfig-adjacent gap's code shape identically to join.c's precedent), but closing the user-visible bug additionally requires a second, larger, currently-unscoped change: something must actually populate primary_host/primary_port on a standby from live heartbeat/AppendEntries/ctrlhello traffic (touches ha.c, which has other work in flight this session was told not to touch), OR xceadmin needs a bug-854-style client-side master-ring rotation fallback for schedAdmin/upgradeConfig (mirroring xcecli_ctrlstatus_primary's fix) so an empty-redirect standby reply drives a retry against the next configured master instead of surfacing the bare rejection. See bug-1701 for the dedicated tracking of the underlying data-gap finding.
UPDATE (2026-08-26): bug-1701 is now FIXED (local peers_cfg lookup in handle_ctrlhello, src/controller/config_repl.c:2170-2246 -- see bug-1701's own entry for the full fix writeup) and LIVE-VERIFIED end to end, closing the gap that made this bug's fix a live no-op. Re-ran the exact failing scenario from this bug's own live-verification section: fresh 3-node cluster, xceadmin scheduler record status --controller <standby> now prints redirected: <standby host:port> -> <primary host:port> on stderr and the real recording status on stdout, exit 0 -- confirmed against both standbys in the cluster and again after a re-election changed which node was primary. This bug's own code change (myreq_handler's fill_redirect_reply() call on -XCE_ERR_STANDBY, cluster.c ~410-433) required NO further modification -- it was correct all along; it only needed a real, populated ha->primary_host/primary_port to redirect to, which bug-1701 now supplies. Status: FIXED, LIVE-VERIFIED.
Original PENDING fix text (superseded by the above, kept for history): PENDING. Minimal, single-call-site fix: in myreq_handler's generic ha_dispatch_gate rejection path (cluster.c:406-414), when the rejection reason is specifically -XCE_ERR_STANDBY (i.e. this node knows or may know who the primary is) call fill_redirect_reply(&reply, &c->ha) before xce_reply(), exactly mirroring OPCode_join's existing handle_worker_join()-adjacent usage (join.c:277). This fixes OPCode_schedAdmin AND the already-documented OPCode_upgradeConfig gap simultaneously, with ZERO xceadmin/client-side changes needed -- the existing follow_redirects() mechanism will start working immediately once the server populates the fields. Must verify: (a) ha_state_t.primary_host/primary_port are populated/valid on a CTRL_STANDBY node that has learned of a live primary (the common case) vs. a node that genuinely does not know any primary yet (mid-election, or has never heard from one) -- fill_redirect_reply() unconditionally aliases these fields with no NULL/empty guard, so confirm what a standby's primary_host looks like before any primary is ever discovered (likely NULL/empty, in which case the reply's redirectHost would encode as empty/absent and the client's existing if host and port guard already handles this correctly as 'no redirect' -- but verify, don't assume); (b) add a regression test proving a schedAdmin request to a standby-with-known-primary now receives a populated redirect reply (mirroring test_join_redirect.c's existing pattern, test_redirect_reply_carries_primary_address) and a live xceadmin CLI test (extending test_e2e_live_ha_push.py's pattern) proving xceadmin scheduler record status against ANY of the 3 nodes now transparently reaches the primary and succeeds.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1698, bug-854, bug-1701