Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
5.3 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | fix_commits | legacy | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-851 | fixed | 2026-09-24 | controller | data-loss | 2026-06-21T13:30:00.000Z | 2026-09-19T09:00:00Z | 2 |
|
|
|
|
true |
bug-851: CHAOTIC leadership churn (3x kill-the-primary) causes SILENT committed-prefix DIVERGENCE: all 3…
Description
CHAOTIC leadership churn (3x kill-the-primary) causes SILENT committed-prefix DIVERGENCE: all 3 nodes report committedLsn=54 with lsn range 1..54 / 0 gaps / CRC intact, but WAL CONTENT diverges at LSN 45-54 (ctrl-1/ctrl-2/ctrl-3 each hold DIFFERENT jobs at the same LSNs; fps bfeb26ad/216af434/865304ec). Violates Raft committed-prefix invariant (a committed entry must be byte-identical on every member). Originally seen as a liveness 'commit stall'; the deeper defect is divergence + acked-job loss under churn.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
ROOT-CAUSED 2026-06-21 (owned-cluster repro, see .kiro/specs/ha-commit-durability/repro-851/). THREE compounding defects: (1) Election term not monotonic across cluster: term = locally-persisted+1, PreVote does NOT persist/propagate ('PreVote round ... no persist'); under chaotic kills a leader wins term N, appends own-term noop+jobs, and is killed BEFORE term N propagates to a majority; a restarted node recovers a STALE term and elects at the SAME N+1 => TWO leaders share a term (logs: term3 led by ctrl-3@08:40 AND ctrl-1@08:47:39; term4 led by ctrl-2@08:47:52 AND ctrl-3@08:48:04), each appending its own-term noop at the SAME lsn 45. (2) WAL entry term field is DEAD (always 0): wal_entry_hdr_t.term exists (wal_types.h:30) but writer does hdr.term=p->term (persist.c:588/726/1040) and p->term is NEVER assigned from ha->term anywhere in the controller (zero matches) => every entry persists term=0; waldump confirms term=0 cluster-wide. (3) No (term,lsn) reconcile: with no usable per-entry term, ha_convergence.c reconcile cannot Raft-log-match, so a divergent same-lsn suffix (45@term3 vs 45@term4) looks already-present and is never truncated; ha_compute_safe_committed_lsn then advances committed 'from 1 granted voter' over content other members never received. NOTE: a CLEAN single failover fully converges (all WALs identical 1..22, 848/849/850 validated) — only CHAOTIC churn triggers it.
Fix
FIXED AND VERIFIED LIVE -- three distinct contributing root causes were found and fixed, all owned by ha-term-safety (spec.json validate_impl decision: GO, 2026-07-17): (1) Term propagation -- repl_broadcast (persist.c:747, the live replicator) built its wire entry without setting entry.term, so live traffic wire-encoded term=0 even though the primary's own header had the real term (same root cause as bug-990, fixed together in task 3.4: entry.term=hdr->term now carried through the live send+intake path, term-0 legacy direct-apply retired). 175/175 unit tests, live-confirmed on real fs. (2) Election-Safety violation (bug-1095/bug-1096, the 'bug-851 family', found by the raft-sim-conformance oracle): raft_on_vote_reply in libxce's raft.c double-counted duplicate vote replies and never dropped stale-term replies, allowing two leaders at the same term -- fixed with a per-peer vote-reply dedup + stale-term drop, unit-tested at both the raft-sim and controller level. (3) Premature election-time truncation (bug-828, closely related): ha_enter_real_election was truncating the WAL to a stale last_known_committed_lsn before computing the safe commit point, destroying already-durable committed entries -- fixed (commit b77d412), live-verified: clean failover preserves committed jobs, walcompare committed-prefix identical across all 3 members across multiple runs. ha-term-safety's GO decision explicitly cites a fresh 6-round LIVE chaotic-multi-kill soak (bug-851's exact repro scenario) passing with zero committed divergence/job loss -- superseding an earlier in-task note (task 5.4) that had called this env-blocked before that later live run completed.
[Original entry] NOT FIXED (HA-core + persistence => full spec workflow per CLAUDE.md, not a quick patch). Keystone lever: stamp hdr.term from live ha->term on the primary append path + carry term through the repl envelope + persist it, THEN make ha_convergence reconcile truncate any local suffix whose (term,lsn) != leader's (standard Raft log-matching). MUST pair with durably-establishing the election term before a new leader acts (no two leaders per term: persist/propagate the won term to a majority before appending the own-term noop, or bump term beyond any (term,lsn) a voter advertises). Route to ha-consistency-invariants / ha-convergence-recovery as a new sub-spec.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-850, bug-849, bug-848, bug-646
Legacy notes
- root_caused: 2026-06-21T09:10:00.000Z