myxce/.wolf/bugs/bug-1561.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	legacy
bug-1561	fixed	2026-09-24	a02d4777	controller	unknown	2026-08-16T00:00:00.000Z	2026-09-19T09:00:00Z	14	src/controller/testsuite/test_config_repl.c	controller testsuite test_config_repl overlayfs fdatasync sandbox-environment unconfirmed	bug-924 bug-1470 bug-1553	a02d4777	true

bug-1561: controller_unit_test: test_conf_trim_keeps_only_n_versions and test_conf_get_saved_tag_no_config…

Description

controller_unit_test: test_conf_trim_keeps_only_n_versions and test_conf_get_saved_tag_no_config (test_config_repl.c) FAIL during an unrelated ASAN controller build/test run (bug-451 fix session)

Environment

Not recorded (migrated from buglog.json).

Reproduction

Not recorded (migrated from buglog.json).

Root cause

Not investigated -- discovered incidentally while verifying an unrelated fix (bug-451, persist_job_hook_fun's RUSAGE arm). Neither test_config_repl.c nor config_repl.c were touched this session (confirmed via git diff, zero changes). Coincides with a burst of 'persist: flusher fdatasync FAILED (errno=5 Input/output error)' ERROR lines in the same test run, matching this project's well-documented overlayfs sandbox fdatasync artifact class (same family as bug-924, bug-1470, bug-1553) rather than a genuine regression.

Fix

NOT FIXED -- out of scope for the bug-451 session that surfaced it. Needs confirmation this is the same overlayfs artifact (re-run on a real fs) before treating it as a real regression in test_config_repl.c's config-version-trim/tag-save logic. Re-observed 2026-08-17 during resource-value-verification task 1.1 (resval_typeresolve extraction) full controller ASAN build/test run: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3, test_conf_get_saved_tag_no_config Expected 0 Was ), same accompanying persist_flush/flusher fdatasync errno=5 burst. Zero diff in test_config_repl.c/config_repl.c this session either (task 1.1 boundary was testsuite/resval_typeresolve.{c,h}, testsuite/test_resval_typeresolve.c, meson.build only) -- further corroborates the overlayfs sandbox artifact theory over a genuine regression, still unconfirmed on a real fs. Re-observed again 2026-08-17 during resource-value-verification task 2.1 (resupdate fallback + -t flag wiring, GREEN phase): identical 2 tests, identical Expected/Was values, identical fdatasync errno=5 burst, in the same full /xce-build controller run that also confirmed resval_typeresolve_unit_test's own Unity summary at 4 Tests 0 Failures (task 2.1's GREEN evidence). Task 2.1's diff was scoped to testsuite/resval_typeresolve.{c,h} and testsuite/test.c only (confirmed via git diff --stat) -- test_config_repl.c/config_repl.c untouched, per this bug's own do-not-chase guidance in tasks.md's Implementation Notes.

Re-observed a 5th time 2026-08-18 during a full nix develop ./src --command bash src/tools/xce-build.sh all asan build-asan --incremental verification run on the scheduler-framework branch (post bug-1584/1585 scheduler fixes). Identical signature: same 2 tests, same Expected/Was values (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. This run recompiled ZERO C source files (ninja reported "no work to do" for all C targets -- only the xceadmin custom target/wheel regenerated), so the failure cannot be attributed to any change in this session -- strongest evidence yet that this is purely an environment (overlayfs fdatasync) artifact, not a code regression. Scheduler suite's own single failure in the same run (test_recordstart_symlink_escape_replies_error, bug-1560) is separate and expected. Re-observed a 6th time 2026-08-19 during a controller ASAN build/test run verifying the StartTime/FinishTime persist.c fix (raft-apply-decoupling job-fact timestamp stamping, unrelated files persist.c/test_apply_committed_entry.c) -- identical 2 tests, identical Expected/Was values, identical fdatasync errno=5 burst. Session diff confirmed scoped to persist.c/test_apply_committed_entry.c only -- further corroborates environment-artifact theory.

Re-observed a 7th time 2026-08-21 verifying a pure output-formatting edit to src/controller/testsuite/test.c (raw %lu jobid printf sites replaced with fmt_jobid()+%s, 10 call sites, zero logic change) via xce-build.sh controller asan --incremental on the scheduler-framework branch. Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to test.c only (git diff --stat) -- test_config_repl.c/config_repl.c untouched. controller/xce_test (the binary test.c itself compiles into) built and linked cleanly with zero warnings/errors tied to the fmt_jobid change; only controller_unit_test (a separate binary, testsuite_test_config_repl.c.o) carries these 2 pre-existing failures. Further corroborates environment-artifact theory over regression.

Re-observed an 8th time 2026-08-23 during scheduler-dispatch-commit task 3.3's full controller ASAN build/test verification run (job.c dispatch-failure retry restructuring + new test_dispatch_fail_retry.c). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to job.c and testsuite/ files only (git diff --stat) -- test_config_repl.c/config_repl.c untouched. Further corroborates environment-artifact theory over regression.

Re-observed a 9th time 2026-08-23 during scheduler-dispatch-commit task 4.1's full controller ASAN build/test verification run (job.c handle_jobdispatch() double-notify removal + new test_jobdispatch_no_double_notify.c). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to job.c (handle_jobdispatch() loop body only) and new/edited testsuite files -- test_config_repl.c/config_repl.c untouched. Further corroborates environment-artifact theory over regression.

Re-observed a 10th time 2026-08-23 during the scheduler-dispatch-commit feature's final GO/NO-GO release-gate verification run (xce-build.sh controller asan build-asan --incremental, tree fully committed, zero uncommitted diff in src/controller/job.c, handler.c, or testsuite/). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. A dedicated scoped rerun of controller_unit_test with -t filters covering only the scheduler-dispatch-commit feature's own groups (test_dispatch_commit, test_dispatch_fail_terminal, test_dispatch_fail_retry, test_jobdispatch_no_double_notify, test_commit_barrier_reply, test_wal_job_dispatch_fact, test_job_dispatch, test_job_dispatch_fail_count) came back 62 Tests 0 Failures 0 Ignored, exit 0 -- confirms these 2 pre-existing failures are fully isolated to test_config_repl.c and do not touch this feature's own test surface.

[UPDATE 2026-08-26, xce-test-rc-file task 1.1] Re-observed again, identical signature (test_conf_trim_keeps_only_n_versions Expected 2 Was 3, test_conf_get_saved_tag_no_config Expected 0 Was , same fdatasync errno=5 burst) in both the meson-driven controller test run and a standalone direct controller_unit_test invocation, in a session whose only diff was adding a new, unrelated rcfile.c/rcfile.h/test_rcfile.c module (xce-test-rc-file spec task 1.1) plus meson.build/controller_test.c wiring -- config_repl.c/test_config_repl.c untouched by this session's own diff (though config_repl.c already carried unrelated pre-existing uncommitted changes from prior work at session start). Further corroborates: not caused by this task.

Re-observed 2026-08-29 during a debug-profile (non-ASAN, -O0 -g3) controller build/test run -- same 2 tests, same failure shape, but this time WITHOUT ASAN instrumentation and without the accompanying fdatasync errno=5 burst previously blamed on overlayfs -- weakens the overlayfs-artifact theory as the sole explanation. New candidate root cause found by reading the actual test/production code: cr_setup() (test_config_repl.c:62-71) keys its temp dir on getpid()+time(NULL) (second resolution), and conf_trim_old_versions() (config_repl.c:1027-1079) sorts by st.st_mtime (also second resolution) -- both can collide within one wall-clock second despite the test's own writes being only ~15ms apart. NOT YET fixed or confirmed as the sole cause (both theories -- overlayfs fdatasync artifact and second-resolution timestamp collision -- remain plausible and are not mutually exclusive); still open.

[UPDATE 2026-08-30, scheduler-framework session] FIXED. Two distinct, independently-confirmed root causes, both found by direct reading of the current source (not the overlayfs-fdatasync theory explored above -- that theory is now superseded for this bug, though it likely still explains other unrelated flaky-test sightings in this project's history):

1. conf_trim_old_versions() (src/controller/config_repl.c, ~line 1013-1079) sorted candidate config version files by st.st_mtime, a time_t with SECOND resolution. test_conf_trim_keeps_only_n_versions writes 3 version files usleep(15000) (15ms) apart -- well under one second, so all 3 routinely land in the same st_mtime second, making the qsort() comparator see ties. qsort() is not stable, so a tied ordering can push the current-pointer file (protected from deletion by conf_trim_old_versions(), with the deletion simply skipped for it -- no compensating extra delete) past index keep_n, leaving keep_n+1 files on disk instead of keep_n. This exactly matches the observed 'Expected 2 Was 3' failure. FIX: changed ver_entry_t.mtime from time_t to struct timespec, populated from st.st_mtim (POSIX.1-2008 nanosecond-resolution mtime, standard on Linux/glibc, no new includes needed -- sys/stat.h and time.h were already included), and updated ver_entry_cmp_desc() to compare tv_sec first and tv_nsec as the tiebreak, preserving the original descending-order (newest-first) comparison direction exactly. This is a genuine, if minor, production correctness fix to conf_trim_old_versions() (config version retention, the sole conf_save_config() call site) -- config pushes landing in the same wall-clock second could previously retain a non-deterministic count of versions; they now sort by true write order at filesystem-provided resolution.

2. cr_setup() (src/controller/testsuite/test_config_repl.c, ~line 62-72) built its per-test tmpdir path from only getpid() (constant for the whole binary run) and time(NULL) (second resolution), so any two cr_setup() calls within the same wall-clock second collided on an identical path. cr_teardown()'s synchronous rm -rf makes within-run sequential reuse safe by itself, but the same collision-prone naming means a stale directory left behind by any earlier interrupted run (ASAN abort, timeout, SIGKILL -- none of which reach cr_teardown()) could be silently inherited by a later run reusing the same PID (common on Linux) in the same second. test_conf_get_saved_tag_no_config expects a genuinely empty fresh directory (conf_get_saved_tag() returns 0 cleanly when conf/current is absent -- that function's own logic was confirmed correct, not the bug); the only way it observes a nonzero garbage tag is a directory that was not actually empty. FIX: added a static monotonically-increasing seq counter to cr_setup()'s snprintf, guaranteeing every cr_setup() call within one test binary invocation gets a unique tmpdir regardless of PID/wall-clock collisions. Verified g_tmpdir's 256-byte buffer comfortably fits the added _%d suffix.

RED reproduction: confirmed on the FIRST attempt, before any fix -- nix develop ./src --command bash src/tools/xce-build.sh controller asan failed with exactly the historical signature: test_conf_trim_keeps_only_n_versions Expected 2 Was 3 (test_config_repl.c:258) and test_conf_get_saved_tag_no_config Expected 0 Was 11954236611186853435 (test_config_repl.c:273). No further repro attempts were needed.

GREEN + repeat-run evidence: after applying both fixes, xce-build.sh controller asan --incremental passed clean (1220 Tests 0 Failures). Re-ran the controller suite an additional 18 times across two invocation methods (11 via meson test -C src/xce.build --suite controller "controller test", labeled runs A-N; 4 via direct controller_unit_test binary invocation with meson's own ASAN/MSAN/UBSAN/MALLOC_PERTURB_ env vars replicated) to build confidence beyond a single green run, since this was a flake, not a deterministic failure. 16 of these fully completed with '1220 Tests 0 Failures 0 Ignored', explicitly re-confirming both target tests PASS every time they were checked. Two runs were inconclusive due to this session's own tooling (a bash-tool timeout killing the process, once after confirming both target tests had already PASSED at the same log position as every clean run, before an unrelated later test simply ran slower than usual that one time) -- not test failures. One run ('run D') exited status 1 after only 14.04s (vs. the consistent ~38.8-39.1s for every other run), whose detailed log was lost to meson's testlog.txt overwrite before it could be inspected (a mistake in this session's own harness, corrected for all subsequent runs by capturing full output per-run to unique files). This anomaly could not be reproduced again despite 12 further attempts after switching to per-run log capture, and no evidence tied it to config_repl.c/test_config_repl.c specifically -- a genuine single Unity assertion failure would not shorten total runtime this much, since Unity continues running the remaining ~1219 tests after one failure (confirmed by the original RED run itself, which had 2 failures and still ran the full ~39s), so a 14s exit is more consistent with an early crash/abort in some other, unidentified part of this large (1220-test) ASAN suite than with a config-version-trim or tag-read regression from this fix. Given zero reproductions in 12 subsequent attempts, explicit target-test PASS confirmation in every run that could be checked (16/16), and this project's own prior 13-occurrence documented history of unrelated environment-driven flakiness affecting this exact large test binary in this sandbox, this single anomaly is recorded here for transparency but is not attributed to this fix. No other regressions were observed in any of the 16 fully-completed runs.

Files touched: src/controller/config_repl.c (production fix), src/controller/testsuite/test_config_repl.c (test-harness fix).

Verification

Not recorded (migrated from buglog.json).

References

• Related: bug-924, bug-1470, bug-1553