Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
4.4 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1487 | fixed | 2026-09-24 | unknown | unknown | 2026-07-31T04:50:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1487: 30-round chaotic HA soak (run-ha-soak.sh --chaotic 30): committed_job_loss=1 at round 25/30…
Description
30-round chaotic HA soak (run-ha-soak.sh --chaotic 30): committed_job_loss=1 at round 25/30 (VERDICT result=FAIL). expect-job reported job 0-6-1785473261-216067 not found on newly-elected primary ctrl-2 (term=27) immediately after the round-25 kill of ctrl-1 (term=26). committed_divergence stayed 0 for the entire run; the job (and all others) was confirmed present on every subsequent verify (rounds 26-30) and the final post-run sweep of all 93 tracked jobs.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
NOT a controller bug and NOT genuinely-uncommitted-at-kill-time -- test-harness false-positive, root-caused with log/timestamp evidence from the preserved run (/home/claude/tmp/xce-ha-chaos30-run1/logs/ctrl-{1,2,3}.log). ctrl-1 (primary, term=26) majority-committed all 3 round-25 jobs (lsn 101,102,103; self + ctrl-2 ack) between 04:47:41.839 and 04:47:42.052 -- confirmed via commit_advance_locked log lines -- moments before being killed at 04:47:42+. ctrl-2 was elected primary at term=27 at 04:47:52.470 with ha_compute_safe_committed_lsn reporting safe_committed_lsn=99 "(from 2 granted voters)" -- the conservative Raft floor, since ctrl-3 (the only other reachable voter) was itself mid-restart/catch-up from round 25s own overlapping-kill design and had not yet caught up past lsn 99. Per the documented Raft Figure-8 safety mechanism (ha_arm_own_term_gate, ha.c:4997-5046: "the universal commit barrier still requires per-entry majority of a new-term entry"), a newly-elected primary cannot fold a PRIOR-term entry into its own committed_lsn watermark merely because it holds it durably -- it must first get a fresh majority ack, under its OWN term, for its own-term WAL_NOOP (armed at promotion). ctrl-2s committed_lsn stayed at 100 (excluding lsn 101-103) until ctrl-3s WAL catch-up stream let it ack the term-27 noop at lsn=104, at 04:47:58.702 -- 6.232s after promotion (commit_advance_locked: "committed_lsn 100 -> 104"). expect-job (joblist query) reads c->jobmgr, which the apply pump only populates for entries that have reached this committed_lsn watermark -- so the job was genuinely invisible, by design, for that 6.232s window. run-ha-soak.shs verify_classify() (chaotic mode) runs its expect-job check immediately once wait_for_new_primary detects the new primary is SERVING at a higher term -- unlike verify_no_loss (sequential mode; already carries a 3-attempt/2s-settle retry from bug-1422 for a related but distinct client-retry-storm race) and unlike wait_for_full_convergence (which chaotic mode intentionally skips between kills, by design, to land kills inside overlap windows) -- so its single, instantaneous query landed inside the 6.232s barrier window and counted a false LOSS. Comparison: round 24s election (ctrl-1, term=26) closed the identical barrier window in only 167ms because the other voting peer was not mid-catch-up at that moment -- the window duration is directly proportional to how far behind the second required peer is, which chaotic modes overlapping-restart design can occasionally push to several seconds.
Fix
Added a bounded retry (5 attempts, 2s settle between) around the expect-job/verdict-classify check in verify_classify() (.claude/skills/xce-test-ha/run-ha-soak.sh) -- mirrors verify_no_loss/bug-1422s "retry the whole check" precedent, only retrying on a LOSS classification (BUG854/OK still resolve on the first attempt, matching prior semantics). ~8s of total settle margin comfortably covers the measured 6.232s worst-case gap while staying bounded -- a job still missing after 5 attempts still counts as a genuine LOSS, so real committed-job loss detection is unweakened. Verified with a full, fresh 30-round chaotic re-soak (see run2 VERDICT).
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1422, bug-1431, bug-1432