Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
3.4 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-606 | fixed | 2026-09-24 | controller | hang | 2026-06-16T17:46:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-606: After killing the primary (ctrl-1), the two surviving standbys (ctrl-2,ctrl-3) livelock in…
Description
After killing the primary (ctrl-1), the two surviving standbys (ctrl-2,ctrl-3) livelock in synchronized elections: PreVote win → CTRL_ELECTING → become_follower (term bumped) → ~10s → repeat, term climbing unbounded (3→9→∞), no new primary ever elected. Reproduce: deploy 3-node HA, elect primary, submit 3-5 jobs, kill ctrl-1.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
Dueling-candidates split-vote livelock — Raft §5.2 randomized election timeout is missing on the FAILOVER path. Both survivors lose the primary at the same instant; their liveness polls fire on the identical fixed heartbeat_timeout, so they start PreVote+real-election in perfect lock-step (ms-aligned timestamps). Both win PreVote (neither sees a primary, so each grants the other — PreVote cannot break this tie), then both campaign at the same term, each votes for itself and denies the other (quorum 2, only 2 live nodes) → split vote → election times out (fixed election_timeout_ms, raft.c:105, NO jitter) → become_follower, term+1 → repeat forever. The priority/jitter config (election_jitter_max_ms, election_priority_step_ms) is applied ONLY at startup (ha_wait_for_heartbeat_or_timeout); the failover re-election path (liveness_poll_fn → ha_on_primary_ctrl_fd_rdhup → PreVote → ha_start_election_async → raft_start_election) has no de-synchronization. raft_on_leader_lost DOES have a priority stagger (raft.c:421) but the failover loop does not go through it.
Fix
Added ha_election_stagger_ms() (ha.c/ha.h): per-node failover delay = my_priority*election_priority_step_ms + random[0,election_jitter_max_ms], re-randomized each call (mirrors the startup stagger). ha_begin_failover() now schedules the PreVote start via delaymgr after that stagger (production only; c==NULL/no-delaymgr stays synchronous so existing tests are unaffected). The delaymgr callback ha_failover_stagger_fn guards with role==STANDBY && ha_heartbeat_is_fresh → skips PreVote if a primary connected during the stagger. Effect: two survivors no longer campaign in lock-step; the higher-priority survivor wins and its heartbeats reset the other. TDD: test_election_stagger_desyncs_by_priority (bounds + step>jitter ordering). VERIFIED on kill-primary repro: ctrl-2 (stagger 630ms) won → PRIMARY term=1; ctrl-3 (stagger 1072ms) received ctrl-2's ctrlhello first → STANDBY, then its stagger timer hit the guard and skipped PreVote ('a primary is alive — skipping PreVote'). No term inflation, no livelock. NOTE: optional raft.c:105 give-up-watchdog randomization NOT needed — the failover-start stagger alone resolves it; left as future defense-in-depth.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-605