diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 551f8e39..8dc5d243 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -7847,7 +7847,7 @@ "error_message": "controller_unit_test: ~63 'Expected Non-NULL' + test_config_repl catchup FAILs (Expected 0 Was -11 EAGAIN) during ha-term-safety 1.3 validation", "file": "environment (/ overlay filesystem)", "root_cause": "Disk filled to 100% (7466 leftover /tmp/xce_* WAL dirs from repeated ASAN test runs). make_fresh_persist's persist_init mkdir_p hit ENOSPC -> returned NULL -> cascade of 'Expected Non-NULL'. Also overlayfs returned fdatasync EIO under disk pressure. NOT a code regression: baseline full suite passes; test_config_repl catchup tests are order-dependent and fail in isolation on baseline too.", - "fix": "rm -rf /tmp/xce_* freed 327GB; full controller suite then Ok:1 Fail:0. For future runs: clean /tmp/xce_* between heavy ASAN test cycles.", + "fix": "rm -rf /tmp/xce_* freed 327GB; full controller suite then Ok:1 Fail:0. For future runs: clean /tmp/xce_* between heavy ASAN test cycles.\n\n[UPDATE 2026-08-26, xce-test-rc-file task 1.1] Re-observed at much larger scale (14731 leftover /tmp/xce_* dirs, 653.6G, filesystem at 100% used / 0 available) and a MORE SEVERE manifestation than the original 'Expected Non-NULL' assertion failures: mkdtemp() failing inside test_repl_apply.c's make_test_persist_for_apply()/make_test_persist_in() hit ENOSPC, and that pre-existing test file's own hard exit(1) on mkdtemp failure silently killed the ENTIRE controller_unit_test process mid-run (right after test_repl_unit's last test, well before test_repl_apply's own output ever printed) -- exit code 1, no crash/ASAN report, no visible error in the redirected log until grepped for the literal 'mkdtemp failed' stderr line. A leftover orphaned controller_unit_test process (PID from an earlier invocation, blocked in futex_do_wait, presumably stalled retrying a WAL fsync against the full disk) was also found and killed. Fixed for this session via `find /tmp -maxdepth 1 -name 'xce_*' -exec rm -rf {} +` (630G reclaimed, 100% -> 29% used) -- confirmed no live xce process had any of those dirs open before deleting. General lesson: when controller_unit_test dies silently/instantly with exit code 1 and the log just stops mid-file with no crash signature, check `df -h /tmp` and grep for 'mkdtemp failed'/'calloc failed' before assuming a real regression -- this sandbox's /tmp accumulates leftover unit-test mkdtemp() fixture dirs across EVERY session indefinitely (no test file in this suite cleans up its own /tmp/xce_* dirs), so this will keep recurring until either the testsuite adds its own cleanup or something periodically clears /tmp.", "tags": [ "disk-space", "enospc", @@ -7858,8 +7858,8 @@ "ha-term-safety" ], "related_bugs": [], - "occurrences": 1, - "last_seen": "2026-06-22" + "occurrences": 2, + "last_seen": "2026-08-26T00:00:00Z" }, { "id": "bug-925", @@ -16304,7 +16304,7 @@ "error_message": "controller_unit_test: test_conf_trim_keeps_only_n_versions and test_conf_get_saved_tag_no_config (test_config_repl.c) FAIL during an unrelated ASAN controller build/test run (bug-451 fix session)", "file": "src/controller/testsuite/test_config_repl.c", "root_cause": "Not investigated -- discovered incidentally while verifying an unrelated fix (bug-451, persist_job_hook_fun's RUSAGE arm). Neither test_config_repl.c nor config_repl.c were touched this session (confirmed via git diff, zero changes). Coincides with a burst of 'persist: flusher fdatasync FAILED (errno=5 Input/output error)' ERROR lines in the same test run, matching this project's well-documented overlayfs sandbox fdatasync artifact class (same family as bug-924, bug-1470, bug-1553) rather than a genuine regression.", - "fix": "NOT FIXED -- out of scope for the bug-451 session that surfaced it. Needs confirmation this is the same overlayfs artifact (re-run on a real fs) before treating it as a real regression in test_config_repl.c's config-version-trim/tag-save logic. Re-observed 2026-08-17 during resource-value-verification task 1.1 (resval_typeresolve extraction) full controller ASAN build/test run: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3, test_conf_get_saved_tag_no_config Expected 0 Was ), same accompanying persist_flush/flusher fdatasync errno=5 burst. Zero diff in test_config_repl.c/config_repl.c this session either (task 1.1 boundary was testsuite/resval_typeresolve.{c,h}, testsuite/test_resval_typeresolve.c, meson.build only) -- further corroborates the overlayfs sandbox artifact theory over a genuine regression, still unconfirmed on a real fs. Re-observed again 2026-08-17 during resource-value-verification task 2.1 (resupdate fallback + -t flag wiring, GREEN phase): identical 2 tests, identical Expected/Was values, identical fdatasync errno=5 burst, in the same full `/xce-build controller` run that also confirmed resval_typeresolve_unit_test's own Unity summary at 4 Tests 0 Failures (task 2.1's GREEN evidence). Task 2.1's diff was scoped to testsuite/resval_typeresolve.{c,h} and testsuite/test.c only (confirmed via git diff --stat) -- test_config_repl.c/config_repl.c untouched, per this bug's own do-not-chase guidance in tasks.md's Implementation Notes.\n\nRe-observed a 5th time 2026-08-18 during a full `nix develop ./src --command bash src/tools/xce-build.sh all asan build-asan --incremental` verification run on the scheduler-framework branch (post bug-1584/1585 scheduler fixes). Identical signature: same 2 tests, same Expected/Was values (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. This run recompiled ZERO C source files (ninja reported \"no work to do\" for all C targets -- only the xceadmin custom target/wheel regenerated), so the failure cannot be attributed to any change in this session -- strongest evidence yet that this is purely an environment (overlayfs fdatasync) artifact, not a code regression. Scheduler suite's own single failure in the same run (test_recordstart_symlink_escape_replies_error, bug-1560) is separate and expected. Re-observed a 6th time 2026-08-19 during a controller ASAN build/test run verifying the StartTime/FinishTime persist.c fix (raft-apply-decoupling job-fact timestamp stamping, unrelated files persist.c/test_apply_committed_entry.c) -- identical 2 tests, identical Expected/Was values, identical fdatasync errno=5 burst. Session diff confirmed scoped to persist.c/test_apply_committed_entry.c only -- further corroborates environment-artifact theory.\n\nRe-observed a 7th time 2026-08-21 verifying a pure output-formatting edit to src/controller/testsuite/test.c (raw %lu jobid printf sites replaced with fmt_jobid()+%s, 10 call sites, zero logic change) via `xce-build.sh controller asan --incremental` on the scheduler-framework branch. Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to test.c only (git diff --stat) -- test_config_repl.c/config_repl.c untouched. controller/xce_test (the binary test.c itself compiles into) built and linked cleanly with zero warnings/errors tied to the fmt_jobid change; only controller_unit_test (a separate binary, testsuite_test_config_repl.c.o) carries these 2 pre-existing failures. Further corroborates environment-artifact theory over regression.\n\nRe-observed an 8th time 2026-08-23 during scheduler-dispatch-commit task 3.3's full controller ASAN build/test verification run (job.c dispatch-failure retry restructuring + new test_dispatch_fail_retry.c). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to job.c and testsuite/ files only (git diff --stat) -- test_config_repl.c/config_repl.c untouched. Further corroborates environment-artifact theory over regression.\n\nRe-observed a 9th time 2026-08-23 during scheduler-dispatch-commit task 4.1's full controller ASAN build/test verification run (job.c handle_jobdispatch() double-notify removal + new test_jobdispatch_no_double_notify.c). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to job.c (handle_jobdispatch() loop body only) and new/edited testsuite files -- test_config_repl.c/config_repl.c untouched. Further corroborates environment-artifact theory over regression.\n\nRe-observed a 10th time 2026-08-23 during the scheduler-dispatch-commit feature's final GO/NO-GO release-gate verification run (`xce-build.sh controller asan build-asan --incremental`, tree fully committed, zero uncommitted diff in src/controller/job.c, handler.c, or testsuite/). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. A dedicated scoped rerun of controller_unit_test with -t filters covering only the scheduler-dispatch-commit feature's own groups (test_dispatch_commit, test_dispatch_fail_terminal, test_dispatch_fail_retry, test_jobdispatch_no_double_notify, test_commit_barrier_reply, test_wal_job_dispatch_fact, test_job_dispatch, test_job_dispatch_fail_count) came back 62 Tests 0 Failures 0 Ignored, exit 0 -- confirms these 2 pre-existing failures are fully isolated to test_config_repl.c and do not touch this feature's own test surface.", + "fix": "NOT FIXED -- out of scope for the bug-451 session that surfaced it. Needs confirmation this is the same overlayfs artifact (re-run on a real fs) before treating it as a real regression in test_config_repl.c's config-version-trim/tag-save logic. Re-observed 2026-08-17 during resource-value-verification task 1.1 (resval_typeresolve extraction) full controller ASAN build/test run: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3, test_conf_get_saved_tag_no_config Expected 0 Was ), same accompanying persist_flush/flusher fdatasync errno=5 burst. Zero diff in test_config_repl.c/config_repl.c this session either (task 1.1 boundary was testsuite/resval_typeresolve.{c,h}, testsuite/test_resval_typeresolve.c, meson.build only) -- further corroborates the overlayfs sandbox artifact theory over a genuine regression, still unconfirmed on a real fs. Re-observed again 2026-08-17 during resource-value-verification task 2.1 (resupdate fallback + -t flag wiring, GREEN phase): identical 2 tests, identical Expected/Was values, identical fdatasync errno=5 burst, in the same full `/xce-build controller` run that also confirmed resval_typeresolve_unit_test's own Unity summary at 4 Tests 0 Failures (task 2.1's GREEN evidence). Task 2.1's diff was scoped to testsuite/resval_typeresolve.{c,h} and testsuite/test.c only (confirmed via git diff --stat) -- test_config_repl.c/config_repl.c untouched, per this bug's own do-not-chase guidance in tasks.md's Implementation Notes.\n\nRe-observed a 5th time 2026-08-18 during a full `nix develop ./src --command bash src/tools/xce-build.sh all asan build-asan --incremental` verification run on the scheduler-framework branch (post bug-1584/1585 scheduler fixes). Identical signature: same 2 tests, same Expected/Was values (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. This run recompiled ZERO C source files (ninja reported \"no work to do\" for all C targets -- only the xceadmin custom target/wheel regenerated), so the failure cannot be attributed to any change in this session -- strongest evidence yet that this is purely an environment (overlayfs fdatasync) artifact, not a code regression. Scheduler suite's own single failure in the same run (test_recordstart_symlink_escape_replies_error, bug-1560) is separate and expected. Re-observed a 6th time 2026-08-19 during a controller ASAN build/test run verifying the StartTime/FinishTime persist.c fix (raft-apply-decoupling job-fact timestamp stamping, unrelated files persist.c/test_apply_committed_entry.c) -- identical 2 tests, identical Expected/Was values, identical fdatasync errno=5 burst. Session diff confirmed scoped to persist.c/test_apply_committed_entry.c only -- further corroborates environment-artifact theory.\n\nRe-observed a 7th time 2026-08-21 verifying a pure output-formatting edit to src/controller/testsuite/test.c (raw %lu jobid printf sites replaced with fmt_jobid()+%s, 10 call sites, zero logic change) via `xce-build.sh controller asan --incremental` on the scheduler-framework branch. Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to test.c only (git diff --stat) -- test_config_repl.c/config_repl.c untouched. controller/xce_test (the binary test.c itself compiles into) built and linked cleanly with zero warnings/errors tied to the fmt_jobid change; only controller_unit_test (a separate binary, testsuite_test_config_repl.c.o) carries these 2 pre-existing failures. Further corroborates environment-artifact theory over regression.\n\nRe-observed an 8th time 2026-08-23 during scheduler-dispatch-commit task 3.3's full controller ASAN build/test verification run (job.c dispatch-failure retry restructuring + new test_dispatch_fail_retry.c). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to job.c and testsuite/ files only (git diff --stat) -- test_config_repl.c/config_repl.c untouched. Further corroborates environment-artifact theory over regression.\n\nRe-observed a 9th time 2026-08-23 during scheduler-dispatch-commit task 4.1's full controller ASAN build/test verification run (job.c handle_jobdispatch() double-notify removal + new test_jobdispatch_no_double_notify.c). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to job.c (handle_jobdispatch() loop body only) and new/edited testsuite files -- test_config_repl.c/config_repl.c untouched. Further corroborates environment-artifact theory over regression.\n\nRe-observed a 10th time 2026-08-23 during the scheduler-dispatch-commit feature's final GO/NO-GO release-gate verification run (`xce-build.sh controller asan build-asan --incremental`, tree fully committed, zero uncommitted diff in src/controller/job.c, handler.c, or testsuite/). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. A dedicated scoped rerun of controller_unit_test with -t filters covering only the scheduler-dispatch-commit feature's own groups (test_dispatch_commit, test_dispatch_fail_terminal, test_dispatch_fail_retry, test_jobdispatch_no_double_notify, test_commit_barrier_reply, test_wal_job_dispatch_fact, test_job_dispatch, test_job_dispatch_fail_count) came back 62 Tests 0 Failures 0 Ignored, exit 0 -- confirms these 2 pre-existing failures are fully isolated to test_config_repl.c and do not touch this feature's own test surface.\n\n[UPDATE 2026-08-26, xce-test-rc-file task 1.1] Re-observed again, identical signature (test_conf_trim_keeps_only_n_versions Expected 2 Was 3, test_conf_get_saved_tag_no_config Expected 0 Was , same fdatasync errno=5 burst) in both the meson-driven controller test run and a standalone direct controller_unit_test invocation, in a session whose only diff was adding a new, unrelated rcfile.c/rcfile.h/test_rcfile.c module (xce-test-rc-file spec task 1.1) plus meson.build/controller_test.c wiring -- config_repl.c/test_config_repl.c untouched by this session's own diff (though config_repl.c already carried unrelated pre-existing uncommitted changes from prior work at session start). Further corroborates: not caused by this task.", "tags": [ "controller", "testsuite", @@ -16320,8 +16320,8 @@ "bug-1470", "bug-1553" ], - "occurrences": 11, - "last_seen": "2026-08-25T00:00:00Z" + "occurrences": 12, + "last_seen": "2026-08-26T00:00:00Z" }, { "id": "bug-912",