Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
44 lines
6 KiB
Markdown
44 lines
6 KiB
Markdown
---
|
|
id: "bug-924"
|
|
status: "fixed"
|
|
status_updated: "2026-09-24"
|
|
status_note: ""
|
|
component: "unknown"
|
|
severity: "unknown"
|
|
timestamp: "2026-06-22"
|
|
last_seen: "2026-09-19T09:00:00Z"
|
|
occurrences: 4
|
|
files: ["environment (/ overlay filesystem)"]
|
|
tags: ["disk-space", "enospc", "overlayfs", "fdatasync-eio", "controller-tests", "flaky", "ha-term-safety", "recurring", "hang", "timeout"]
|
|
related: []
|
|
legacy: true
|
|
---
|
|
|
|
# bug-924: controller_unit_test: ~63 'Expected Non-NULL' + test_config_repl catchup FAILs (Expected 0 Was -11…
|
|
|
|
## Description
|
|
controller_unit_test: ~63 'Expected Non-NULL' + test_config_repl catchup FAILs (Expected 0 Was -11 EAGAIN) during ha-term-safety 1.3 validation
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Root cause
|
|
Disk filled to 100% (7466 leftover /tmp/xce_* WAL dirs from repeated ASAN test runs). make_fresh_persist's persist_init mkdir_p hit ENOSPC -> returned NULL -> cascade of 'Expected Non-NULL'. Also overlayfs returned fdatasync EIO under disk pressure. NOT a code regression: baseline full suite passes; test_config_repl catchup tests are order-dependent and fail in isolation on baseline too.
|
|
|
|
[UPDATE 2026-09-01, worker-rejoin-resync task 7.2] Re-observed a FOURTH time, with a NEW manifestation shape not previously documented: 14367 leftover /tmp/xce_* dirs, 876.7G, filesystem at 99% used / 12.4G available. This time it did NOT surface as 'Expected Non-NULL' assertion cascades or a silent exit(1) -- it surfaced as controller_unit_test/`meson test` simply HANGING (TIMEOUT, killed by signal 15 after 300s) with no error output at all, reproducible across multiple independent invocations (both via `meson test` and the raw binary directly), always stalled inside persist_fini()/fsync_worker's pthread_cond_timedwait/pthread_cond_destroy teardown path (confirmed via gdb `thread apply all bt` on the wedged process -- Thread 2 stuck in fsync_worker's pthread_cond_timedwait at persist.c:847, Thread 1 stuck in persist_fini's pthread_cond_destroy at persist.c:1362), at a DIFFERENT test each time (not always the same call site) -- consistent with overlayfs fdatasync degrading under near-full disk rather than erroring outright, causing the flusher thread to sit in a slow/stuck wait instead of the previously-documented fast ENOSPC/EIO failure. `find /tmp -maxdepth 1 -name 'xce_*' -exec rm -rf {} +` (no live xce process holding any of them open, confirmed via `ps aux` first) freed 612G (99% -> 30% used); the exact same test run (`controller_unit_test -t test_worker_resync`) that had twice hung/timed out at 90s then completed cleanly in under a second. General lesson extended: an unexplained TIMEOUT/hang in this suite (not just an assertion-failure cascade) is an equally likely symptom of this same recurring /tmp exhaustion -- check `df -h /tmp` before spending time bisecting a hang as a suspected new concurrency bug in the code under test.
|
|
|
|
## Fix
|
|
rm -rf /tmp/xce_* freed 327GB; full controller suite then Ok:1 Fail:0. For future runs: clean /tmp/xce_* between heavy ASAN test cycles.
|
|
|
|
[UPDATE 2026-08-26, xce-test-rc-file task 1.1] Re-observed at much larger scale (14731 leftover /tmp/xce_* dirs, 653.6G, filesystem at 100% used / 0 available) and a MORE SEVERE manifestation than the original 'Expected Non-NULL' assertion failures: mkdtemp() failing inside test_repl_apply.c's make_test_persist_for_apply()/make_test_persist_in() hit ENOSPC, and that pre-existing test file's own hard exit(1) on mkdtemp failure silently killed the ENTIRE controller_unit_test process mid-run (right after test_repl_unit's last test, well before test_repl_apply's own output ever printed) -- exit code 1, no crash/ASAN report, no visible error in the redirected log until grepped for the literal 'mkdtemp failed' stderr line. A leftover orphaned controller_unit_test process (PID from an earlier invocation, blocked in futex_do_wait, presumably stalled retrying a WAL fsync against the full disk) was also found and killed. Fixed for this session via `find /tmp -maxdepth 1 -name 'xce_*' -exec rm -rf {} +` (630G reclaimed, 100% -> 29% used) -- confirmed no live xce process had any of those dirs open before deleting. General lesson: when controller_unit_test dies silently/instantly with exit code 1 and the log just stops mid-file with no crash signature, check `df -h /tmp` and grep for 'mkdtemp failed'/'calloc failed' before assuming a real regression -- this sandbox's /tmp accumulates leftover unit-test mkdtemp() fixture dirs across EVERY session indefinitely (no test file in this suite cleans up its own /tmp/xce_* dirs), so this will keep recurring until either the testsuite adds its own cleanup or something periodically clears /tmp.
|
|
|
|
[UPDATE 2026-08-30, bug-1735 verification build] Re-observed a THIRD time, at the largest scale yet: 16238 leftover /tmp/xce_* dirs, 696.1G, filesystem at 100% used / 0 available. This time it manifested as a `controller_unit_test` run reporting `1220 Tests 282 Failures` (mostly test_dispatch_commit.c/test_dispatch_fail_terminal.c/test_dispatch_fail_retry.c/test_jobdispatch_no_double_notify.c 'Expected Non-NULL' -- the exact same fixture-setup-ENOSPC cascade signature as both prior occurrences, on files wholly unrelated to whatever the run was actually verifying) plus a meson-internal `OSError: [Errno 28] No space left on device` while serializing testlog.json/testlog.junit.xml (corrupting those log artifacts, though the console summary line survived). No live xce process was running at the time (`ps aux` confirmed clean) so `find /tmp -maxdepth 1 -name 'xce_*' -exec rm -rf {} +` was safe; reclaimed 696.1G (100% -> 22% used). Re-running the full suite after cleanup is required before trusting any failure count from a run that coincided with disk exhaustion. Still true three occurrences in: nothing in this test suite cleans up its own /tmp/xce_* dirs, so this WILL keep recurring -- the durable fix is adding cleanup to the shared test fixture helpers, not another one-off `rm -rf`.
|
|
|
|
## Verification
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## References
|
|
_Not recorded (migrated from buglog.json)._
|