Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
6 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1562 | fixed | 2026-09-24 | controller | crash | 2026-08-17T03:40:00.000Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1562: 3-node HA bootstrap never elects a primary: ctrl-1 (only node with local config) goes completely…
Description
3-node HA bootstrap never elects a primary: ctrl-1 (only node with local config) goes completely silent ~22s after boot -- no crash, no log output, cluster.log shows no further activity even after ctrl-2/ctrl-3 join minutes later. Downstream symptom: worker gets 'SSL_do_handshake: Broken pipe' in a loop and ctrl-2/ctrl-3 log repeated 'incorrect worker idx -1' ERRORs from handle_endpoint_error (harmless noise -- join.c's handle_join_ stamps priv->role=HostRole_worker before the ha_no_primary_known guard can reject the join, so u.idx is still -1 when the rejected connection is torn down).
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
Live-reproduced locally (build.dbg xce_controller, fresh 3-node deploy via deploy-ha.sh, ctrl-1 started ~45s before ctrl-2/ctrl-3 to match the user's actual timing) -- byte-for-byte matches the user's ctrl-1.log. ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c ~8573-8589, added 2026-07-01 commit 0d70d5e4, bug-1310 task 5.10b) returns WITHOUT calling ha_arm_heartbeat_timeout_vtask() when a debounced RDHUP lands within the 2x-election-timeout cooldown window, unlike the function's other two early-return branches: the 5.6 election-in-progress guard (checks role==CTRL_ELECTING || pre_vote_in_progress, i.e. actually verifies a live timer is still running elsewhere) and the 5.2 spurious-suppression branch (explicitly re-arms via ha_arm_heartbeat_timeout_vtask before returning). The 5.10b branch only checks elapsed time since last_failover_ms -- it never checks whether an election is ACTUALLY still in flight. On a lone cold-booting node (n_peers up but none reachable yet), the first PreVote round loses within milliseconds (all connect attempts fail fast -> counted as no-votes), which re-arms the liveness watchdog once (10s). When that watchdog fires again ~10s later, last_failover_ms is still inside the 2x-election-timeout cooldown (observed 9805ms < 10000ms), so it lands in the debounce branch even though no election is in flight anymore -- and returns without re-arming anything. That is the LAST timer ever scheduled in the process: /proc//task/*/wchan confirmed every vtask worker thread idling in futex_do_wait (empty work queue, not deadlocked) and the io_worker thread parked in do_epoll_wait with zero further epoll events (lsof showed only the listen socket + epoll fd, no in-flight connections) for the remainder of the run. Node is permanently stuck CTRL_STANDBY with zero further election attempts -- other config-less nodes correctly wait forever for a primary that will never appear, so the whole cluster is permanently leaderless. Timing-dependent: reproducible only when the solo bootstrap window is long enough for a second liveness-timeout to land inside the cooldown after the first PreVote round already concluded (confirmed NOT reproducible with a tight ~0.5s inter-node start stagger, where a real peer answers before the trap window closes) -- explains why this presented as 'used to work' without any actual regression: bug has been latent since 2026-07-01, only the user's own start timing changed.
Fix
FIXED (TDD RED->GREEN). Added test_rdhup_debounce_still_rearms_liveness_watchdog (test_ha_rdhup_integration.c, next to the sibling test_rdhup_within_cooldown_is_debounced) -- asserts ha->last_activity moves off a sentinel after a debounced RDHUP, the same delaymgr-independent observable test_ha_arm_no_crash_no_thread (test_ha_liveness.c) already uses to prove ha_arm_heartbeat_timeout_vtask ran without a live delaymgr thread. RED confirmed pre-fix (assertion failed with the bug-1562 message). Fix: ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c ~8578-8598) now calls ha_arm_heartbeat_timeout_vtask(ha, c) before returning, mirroring the function's own 5.2 spurious-suppression branch pattern -- safe with c==NULL (unit-test mode) since the arm function stamps ha->last_activity unconditionally before its internal c-null/delaymgr-running check gates only the actual timer scheduling. GREEN: controller_unit_test 1102 tests, 2 Failures (both pre-existing, unrelated overlayfs fdatasync EIO sandbox flakiness in test_config_repl.c -- test_conf_trim_keeps_only_n_versions and test_conf_get_saved_tag_no_config -- confirmed via 'persist: flusher fdatasync FAILED (errno=5 Input/output error)' immediately preceding both in the test log; identical to the cerebrum.md-documented sandbox quirk, present before this change too). LIVE END-TO-END VALIDATION (per this project's live-soak-validation-discipline convention -- unit-green alone is not proof): rebuilt xce_controller (src/xce.build), redeployed the identical 3-node scenario that hung before (ctrl-1 solo 45s, then ctrl-2, then ctrl-3 -- byte-for-byte the same timing that previously froze the process forever). Post-fix ctrl-1.log shows the debounce branch now immediately followed by 'HA: standby liveness poll armed' (previously the log went dead right after the debounce line), the watchdog continuing to retry every ~10s through several more PreVote-loses/debounce cycles while alone, and once ctrl-2/ctrl-3 came online: 'HA FSM (raft): -> CTRL_PRIMARY (term=2, first_boot=0)' -- ctrl-1 successfully elected primary, and both controller/ctrl-2/conf and controller/ctrl-3/conf received the pushed cluster.bin + current config. Full cluster convergence confirmed on the exact scenario that previously froze permanently.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1310