Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
8.6 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1701 | fixed | 2026-09-24 | controller | unknown | 2026-08-26T00:00:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1701: Discovered while implementing/live-verifying bug-1700's fix (myreq_handler's generic dispatch-gate…
Description
Discovered while implementing/live-verifying bug-1700's fix (myreq_handler's generic dispatch-gate rejection now calls fill_redirect_reply() on -XCE_ERR_STANDBY, cluster.c ~410-433): the fix has ZERO effect in a real deployment. xceadmin scheduler record status --controller <standby> against a live 3-node cluster still fails with 'Error: controller returned status -10014' identically before and after the fix, with no 'redirected: ...' hop ever printed.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
ha_state_t.primary_host/primary_port are NEVER WRITTEN by any production code path on a standby -- confirmed by grepping every assignment site across the entire src/ tree (grep -rn primary_host src/ --include=*.c --include=*.h): the only writers are test fixtures (test_join_redirect.c, test_p1_integration.c, test_p3_hardening.c, test_ha_redirect.c, test_ha_role_lifecycle.c, this bug's own new test_sched_admin_dispatch.c tests) manually poking the field for isolated unit tests; no heartbeat/AppendEntries/ctrlhello handler in ha.c ever copies the leader's identity into it on the receiving standby. ctrlstatus.c's own field-sourcing doc comment (lines 37-38, 93-101) already names this explicitly ('the vestigial, never-written ha->primary_host') and documents that ctrlstatus_fill() deliberately derives primaryName from role + voted_for_name instead, NEVER from primary_host, for exactly this reason -- so the gap was known and worked around for ctrlstatus, but not for the redirect-reply mechanism (fill_redirect_reply/redirect_host/redirect_port), which has no equivalent workaround and is the ONLY consumer of this permanently-empty field. bug-854 (2026-06-21, CLOSED/FIXED) independently documented the identical fact while root-causing a related but distinct bug (positional client-side primary discovery for ctrlstatus) and explicitly chose NOT to add a server-side primaryName/primary_host source, fixing that bug entirely client-side (xcecli_ctrlstatus_primary master-ring rotation) instead -- a reasonable call for THAT bug, but it leaves fill_redirect_reply() as a piece of server-side machinery that looks functional (it has a real caller, a real unit test proving its own field-copying logic, and a real capnp wire encode/decode path) yet is fed a field that is permanently empty in every real deployment. Direct consequence: OPCode_join's existing use of fill_redirect_reply() (join.c:277, the very precedent bug-1700's fix was told to mirror) is, by the same reasoning, ALSO non-functional against a live standby in production today -- a worker/scheduler/controller join redirect to the primary silently never happens; nothing in the existing test suite (test_join_redirect.c) catches this because those tests call fill_redirect_reply() directly with a manually-populated ha_state_t, never through a live standby that only knows what production heartbeat/election code actually wrote into it.
Fix
FIXED (2026-08-26) via option (a): a pure LOCAL lookup in handle_ctrlhello() (src/controller/config_repl.c:2170-2246, inside the existing primary_ctrl_fd==-1 idempotency-guarded block, ~2185-2233), NOT a new wire-protocol field -- every node already statically knows every candidate's host:port from boot (c->myconf->peers_cfg, parsed once from --peers). At the exact moment a standby first establishes primary_ctrl_fd (the same instant it 'discovers' its primary), it now matches req->from->name (populated for EVERY incoming request at src/lib/request.c:379, req->from = ep->peer; -- the same name= identity used in --peers name=host:port) against c->myconf->peers_cfg.candidates[] by name (loop pattern mirrors the existing voted_for_name-by-index lookup in ha_init_raft, ha.c:7822-7829) and, on a match, copies that candidate's hostname/port into ha->primary_host/primary_port. No match (real config mismatch) or a NULL req->from/name leaves primary_host/primary_port at their previous value (never wiped) and logs a WARN identifying the unmatched sender name. Locking: ha.h's lock field doc explicitly says 'Protects primary_host/port + voted_for_term/name'; the existing writers of voted_for_name (the other field under the same lock, ha.c:2304-2332 and ha.c:5559-5593) take c->ha.lock around the write, so the new write here does too (pthread_mutex_lock/unlock around the strncpy+port assignment only -- the name-matching loop itself runs outside the lock, matching the narrow critical-section style of the precedent). Types verified by direct read, not assumed: ha_state_t.primary_host is char primary_host[XCE_MAXNAME_LEN] (ha.h:161, XCE_MAXNAME_LEN=256, config.h:24) and primary_port is uint32_t (ha.h:162) -- NOT matching the task brief's assumption that primary_port's type already lined up with xce_candidate_t.port (int, cluster.h:22); the int->uint32_t assignment is a safe implicit widening for any real port value (0-65535), cast added for clarity, no risky truncation. No clear-on-primary-loss needed: ha_on_primary_ctrl_fd_rdhup (ha.c, read-only verified, not modified) resets primary_ctrl_fd back to -1 on genuine primary loss, so the next real ctrlhello (from the same or a new primary) re-satisfies the same idempotency guard and naturally re-runs this lookup -- confirmed by reading ha.c:8594-8680, no explicit clear required. RED/GREEN proof: added test_handle_ctrlhello_populates_primary_host_from_known_peer (FAILED pre-fix: "Expected '10.0.1.7' Was ''"; PASSED post-fix) and test_handle_ctrlhello_unknown_peer_leaves_primary_host_unchanged (edge case: unmatched sender name -- no crash, primary_host/primary_port stay at fresh-node empty/zero, primary_ctrl_fd bookkeeping still proceeds) to src/controller/testsuite/test_ha_connection_reuse.c (same file as the pre-existing test_handle_ctrlhello_sets_primary_ctrl_fd, which continues to pass unchanged, proving the idempotency behavior for primary_ctrl_fd itself is untouched). Full controller_unit_test: 1202 Tests, 2 Failures both pre- and post-fix -- exactly the pre-existing bug-1561 overlayfs pair (test_conf_trim_keeps_only_n_versions, test_conf_get_saved_tag_no_config), zero new failures. LIVE END-TO-END VERIFIED -- the exact scenario bug-1700 failed on: deployed two independent fresh 3-node clusters (deploy-ha.sh) on git-stash-isolated before/after controller binaries (stashed only config_repl.c + its test file, leaving other in-flight ha.c/job.c/persist.c work untouched). BEFORE (pre-fix binary): xceadmin scheduler record status --controller localhost:29992 (a standby) -> Error: controller returned status -10014, exit 1, no redirect line -- reproduces bug-1700's original live-verification failure exactly. AFTER (post-fix binary, fresh deploy + also re-verified after clearing controller data to force a different node to win election): xceadmin scheduler record status --controller <either standby> -> stderr prints redirected: localhost:19991 -> localhost:19992 (confirmed via separate stdout/stderr capture: the redirect line is genuinely on stderr, the status body on stdout, matching xceadmin.discovery.follow_redirects()'s documented behavior), stdout prints the real recording status (recording: no, etc.), exit 0 -- verified against BOTH standbys in the cluster and re-verified after a re-election flipped which node was primary (ctrl-2 elected instead of ctrl-1), proving the fix is dynamic/general, not coincidentally tied to one node. This unblocks bug-1700's fix, which was otherwise correct and complete but fed by this permanently-empty field -- see bug-1700's own entry for its updated live-verified confirmation. OPCode_join's pre-existing fill_redirect_reply() usage (join.c:277) is also now unblocked by the same fix (not separately re-verified live in this session, but shares the identical previously-broken data source).
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1700, bug-854