Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
38 lines
4.8 KiB
Markdown
38 lines
4.8 KiB
Markdown
---
|
|
id: "bug-1563"
|
|
status: "fixed"
|
|
status_updated: "2026-09-24"
|
|
status_note: ""
|
|
component: "controller"
|
|
severity: "unknown"
|
|
timestamp: "2026-08-17T04:59:00.000Z"
|
|
last_seen: "2026-09-19T09:00:00Z"
|
|
occurrences: 1
|
|
files: ["src/controller/persist.c (fsync_worker)"]
|
|
tags: ["persist", "fsync", "livelock", "busy-loop", "backoff", "durability", "chaotic-soak", "xce-test-ha"]
|
|
related: ["bug-661"]
|
|
legacy: true
|
|
---
|
|
|
|
# bug-1563: 40-round chaotic /xce-test-ha soak: ctrl-2's own log shows 309,671 'persist: flusher fdatasync…
|
|
|
|
## Description
|
|
40-round chaotic /xce-test-ha soak: ctrl-2's own log shows 309,671 'persist: flusher fdatasync FAILED (errno=22 Invalid argument) -- durable_lsn held at 163 (tip=164 NOT made durable)' ERROR lines, 706 of them landing within a single 705ms window (04:52:54.000-04:52:54.705) on the freshly-promoted primary -- a sustained sync failure the flusher never recovered from or backed off on for the rest of the run. The soak's own verdict still reports PASS (committed_divergence=0, committed_job_loss=0) because the other two standbys' acks alone satisfy majority and the bug-924 fdatasync-error exclusion swallows the count -- but this specific pattern (a stuck LSN, EINVAL not the documented EIO, hundreds of retries per second) is a distinct, real defect from the known overlayfs sandbox quirk, not just noise.
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Root cause
|
|
fsync_worker's loop (persist.c) only calls pthread_cond_timedwait (its ONLY throttle/backoff mechanism) when flusher_work_pending_locked(p) is false. But that predicate is purely 'durable_lsn < tip' -- it cannot distinguish 'genuine fresh work just landed' from 'still stuck from the previous failed attempt', because flusher_flush_once()'s fail-loud gate (correct, unchanged) deliberately never advances durable_lsn on a failed fdatasync. Once fdatasync starts failing with no new appends arriving (a standby/primary transition mid-chaotic-kill with a broken local fd, or any sustained real disk error in production), durable_lsn < tip stays true forever, so 'pending' is permanently true and the loop skips the wait on every single iteration -- flusher_flush_once (and thus fdatasync) gets called as fast as the CPU/lock overhead allows, pinning one core and flooding the log. Confirmed by direct code read of flusher_work_pending_locked (dur < tip, no failure-awareness) and fsync_worker's loop structure (wait only in the !pending branch), and reproduced deterministically in a unit test (test_force_fsync_fail seam) before any live soak was re-run.
|
|
|
|
## Fix
|
|
FIXED (TDD RED->GREEN). Added test_flusher_flush_attempt_count (persist.h, XCE_TESTING-only atomic counter, incremented once per flusher_flush_once() call) and test_flusher_backs_off_on_sustained_fsync_failure (test_term_safety_adapter.c): forces every fdatasync to fail via the existing test_force_fsync_fail seam on a PRIMARY persist_t with wal_sync_ms=20, appends one entry (primary's append path defers all sync to the async flusher, unlike a standby's inline persist-before-ACK sync -- see test_on_persist_entry_failure_propagates_no_ack), waits 300ms, and asserts attempt count stays under 60 (~15 expected at a 20ms cadence). RED confirmed pre-fix (assertion failed -- unbounded spin). Fix: fsync_worker's loop (persist.c) now also waits (same pthread_cond_timedwait/wal_sync_ms cadence the idle path already uses, no new tunable) when the just-completed flush WAS pending but made no progress (after == before) -- i.e. specifically the failure case, never the legitimate 'nothing to sync' idle case (pending was false there, left untouched) nor a successful flush (after > before, still loops immediately to catch a fresh append with zero latency, unchanged fast path). GREEN: test_flusher_backs_off_on_sustained_fsync_failure passes with attempts=15 (was unbounded/thousands pre-fix in the same 300ms window). Full controller_unit_test: 1103 tests, 2 failures -- both pre-existing, unrelated overlayfs fdatasync EIO sandbox flakiness in test_config_repl.c (test_conf_trim_keeps_only_n_versions, test_conf_get_saved_tag_no_config), confirmed unchanged from before this fix. LIVE VALIDATION: re-ran the 40-round chaotic /xce-test-ha soak that originally surfaced this, with the rebuilt binary -- excluded_bug924_fdatasync dropped from 304,689 to 0 in the re-run (VERDICT mode=chaotic rounds=40 jobs=123 committed_divergence=0 committed_job_loss=0 result=PASS). Also re-ran the 40-round sequential soak for completeness (PASS: 40 rounds, 123 jobs, zero job loss, all 3 nodes led) -- unaffected by this change as expected, confirming no regression. Both full 40-round soaks (sequential and chaotic) now pass cleanly end to end.
|
|
|
|
## Verification
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## References
|
|
- Related: bug-661
|