lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

test(worker-rejoin-resync): task 7.2 - failover-during-job recovery

Browse source 
Add test_failover_during_job_recovers_lost_finish, a focused proof of
the feature's own original motivating scenario (bug-1741): a job
dispatched and RUN with no finish fact ever committed anywhere
(modeling a lost fire-and-forget finish notify), recovered to DONE
with the worker-reported finish time - not stuck in RUN - via the
real rejoin/query/report/apply chain. Test-only, no production
changes; reuses task 7.1's helper infrastructure rather than
duplicating its broader four-outcome coverage.

Updates the existing bug-924 (/tmp exhaustion / flaky teardown) entry
with a 5th occurrence hit during validation, unrelated to this task's
own change.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
Rongsong Shen 2026-09-01 23:19:15 +00:00
parent 8650cfd9ba
commit bceca0c990
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
3 changed files with 438 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

10
.wolf/buglog.json
View file

@ -7846,7 +7846,7 @@
"timestamp": "2026-06-22",
"error_message": "controller_unit_test: ~63 'Expected Non-NULL' + test_config_repl catchup FAILs (Expected 0 Was -11 EAGAIN) during ha-term-safety 1.3 validation",
"file": "environment (/ overlay filesystem)",
"root_cause": "Disk filled to 100% (7466 leftover /tmp/xce_* WAL dirs from repeated ASAN test runs). make_fresh_persist's persist_init mkdir_p hit ENOSPC -> returned NULL -> cascade of 'Expected Non-NULL'. Also overlayfs returned fdatasync EIO under disk pressure. NOT a code regression: baseline full suite passes; test_config_repl catchup tests are order-dependent and fail in isolation on baseline too.",
"root_cause": "Disk filled to 100% (7466 leftover /tmp/xce_* WAL dirs from repeated ASAN test runs). make_fresh_persist's persist_init mkdir_p hit ENOSPC -> returned NULL -> cascade of 'Expected Non-NULL'. Also overlayfs returned fdatasync EIO under disk pressure. NOT a code regression: baseline full suite passes; test_config_repl catchup tests are order-dependent and fail in isolation on baseline too.\n\n[UPDATE 2026-09-01, worker-rejoin-resync task 7.2] Re-observed a FOURTH time, with a NEW manifestation shape not previously documented: 14367 leftover /tmp/xce_* dirs, 876.7G, filesystem at 99% used / 12.4G available. This time it did NOT surface as 'Expected Non-NULL' assertion cascades or a silent exit(1) -- it surfaced as controller_unit_test/`meson test` simply HANGING (TIMEOUT, killed by signal 15 after 300s) with no error output at all, reproducible across multiple independent invocations (both via `meson test` and the raw binary directly), always stalled inside persist_fini()/fsync_worker's pthread_cond_timedwait/pthread_cond_destroy teardown path (confirmed via gdb `thread apply all bt` on the wedged process -- Thread 2 stuck in fsync_worker's pthread_cond_timedwait at persist.c:847, Thread 1 stuck in persist_fini's pthread_cond_destroy at persist.c:1362), at a DIFFERENT test each time (not always the same call site) -- consistent with overlayfs fdatasync degrading under near-full disk rather than erroring outright, causing the flusher thread to sit in a slow/stuck wait instead of the previously-documented fast ENOSPC/EIO failure. `find /tmp -maxdepth 1 -name 'xce_*' -exec rm -rf {} +` (no live xce process holding any of them open, confirmed via `ps aux` first) freed 612G (99% -> 30% used); the exact same test run (`controller_unit_test -t test_worker_resync`) that had twice hung/timed out at 90s then completed cleanly in under a second. General lesson extended: an unexplained TIMEOUT/hang in this suite (not just an assertion-failure cascade) is an equally likely symptom of this same recurring /tmp exhaustion -- check `df -h /tmp` before spending time bisecting a hang as a suspected new concurrency bug in the code under test.",
"fix": "rm -rf /tmp/xce_* freed 327GB; full controller suite then Ok:1 Fail:0. For future runs: clean /tmp/xce_* between heavy ASAN test cycles.\n\n[UPDATE 2026-08-26, xce-test-rc-file task 1.1] Re-observed at much larger scale (14731 leftover /tmp/xce_* dirs, 653.6G, filesystem at 100% used / 0 available) and a MORE SEVERE manifestation than the original 'Expected Non-NULL' assertion failures: mkdtemp() failing inside test_repl_apply.c's make_test_persist_for_apply()/make_test_persist_in() hit ENOSPC, and that pre-existing test file's own hard exit(1) on mkdtemp failure silently killed the ENTIRE controller_unit_test process mid-run (right after test_repl_unit's last test, well before test_repl_apply's own output ever printed) -- exit code 1, no crash/ASAN report, no visible error in the redirected log until grepped for the literal 'mkdtemp failed' stderr line. A leftover orphaned controller_unit_test process (PID from an earlier invocation, blocked in futex_do_wait, presumably stalled retrying a WAL fsync against the full disk) was also found and killed. Fixed for this session via `find /tmp -maxdepth 1 -name 'xce_*' -exec rm -rf {} +` (630G reclaimed, 100% -> 29% used) -- confirmed no live xce process had any of those dirs open before deleting. General lesson: when controller_unit_test dies silently/instantly with exit code 1 and the log just stops mid-file with no crash signature, check `df -h /tmp` and grep for 'mkdtemp failed'/'calloc failed' before assuming a real regression -- this sandbox's /tmp accumulates leftover unit-test mkdtemp() fixture dirs across EVERY session indefinitely (no test file in this suite cleans up its own /tmp/xce_* dirs), so this will keep recurring until either the testsuite adds its own cleanup or something periodically clears /tmp.\n\n[UPDATE 2026-08-30, bug-1735 verification build] Re-observed a THIRD time, at the largest scale yet: 16238 leftover /tmp/xce_* dirs, 696.1G, filesystem at 100% used / 0 available. This time it manifested as a `controller_unit_test` run reporting `1220 Tests 282 Failures` (mostly test_dispatch_commit.c/test_dispatch_fail_terminal.c/test_dispatch_fail_retry.c/test_jobdispatch_no_double_notify.c 'Expected Non-NULL' -- the exact same fixture-setup-ENOSPC cascade signature as both prior occurrences, on files wholly unrelated to whatever the run was actually verifying) plus a meson-internal `OSError: [Errno 28] No space left on device` while serializing testlog.json/testlog.junit.xml (corrupting those log artifacts, though the console summary line survived). No live xce process was running at the time (`ps aux` confirmed clean) so `find /tmp -maxdepth 1 -name 'xce_*' -exec rm -rf {} +` was safe; reclaimed 696.1G (100% -> 22% used). Re-running the full suite after cleanup is required before trusting any failure count from a run that coincided with disk exhaustion. Still true three occurrences in: nothing in this test suite cleans up its own /tmp/xce_* dirs, so this WILL keep recurring -- the durable fix is adding cleanup to the shared test fixture helpers, not another one-off `rm -rf`.",
"tags": [
"disk-space",
@ -7856,11 +7856,13 @@
"controller-tests",
"flaky",
"ha-term-safety",
"recurring"
"recurring",
"hang",
"timeout"
],
"related_bugs": [],
"occurrences": 3,
"last_seen": "2026-08-30T06:24:00"
"occurrences": 4,
"last_seen": "2026-09-01T22:18:00"
},
{
"id": "bug-925",