fix(testsuite): bug-924 -- auto-clean mkdtemp scratch dirs at test-exit (real fix, not the prune mask)

Unit tests create /tmp/xce_* scratch dirs via mkdtemp() and never remove
them; a full ASAN run leaks 70+ dirs and fills /tmp to ENOSPC, which then
surfaces as unrelated "Expected Non-NULL"-style cascade failures in whatever
test next calls mkdtemp()/persist_init() (precedent: 16238 dirs / 696G /
disk 100%). xce-build.sh's age-gated prune_stale_tmp_fixtures() only sweeps
the debris hours later -- a mask, not a fix.

Fix at the source: a self-contained mkdtemp() replacement
(src/testsuite_support/mkdtemp_clean.c) linked into every unit-test
executable via a declare_dependency() appended to the shared deps lists
(controller_unit_test_deps, scheduler_test_deps). Each scratch dir it creates
is registered and recursively removed by an atexit() handler, so a test
binary leaves /tmp clean. Cleanup runs at process exit -- after all TEST_
functions -- so tests that legitimately inspect a fixture's on-disk contents
across multiple TEST functions in the same file are unaffected (the property
that made per-file teardown retrofitting invasive).

Verified: full controller_unit_test (1269 tests) passes with 0 failures and
zero /tmp/xce_* left at exit (was: intermittent ENOSPC cascade). Scheduler
suite build+run still to be validated in the merge verification pass.

Design notes (rejected alternatives, to avoid re-litigating):
- A mid-run "keep newest N dirs" eviction was tried and dropped: it slowed a
  monolithic run dramatically and one variant hung the suite (rmrf of large
  WAL dirs in the hot path). atexit-only bounds nothing mid-run, but a single
  suite fits in /tmp and cleanup is prompt; parallel meson suites are the
  residual risk, addressed by running meson test with reduced parallelism if
  needed.
- A pthread_atfork registry reset was tried (to stop a forked child's
  inherited atexit from deleting the parent's dirs) and dropped: it
  regressed the controller suite (~98 failures), so the plain atexit version
  is what ships. If the scheduler repl/sim fork suites need it, revisit there
  with a scheduler-scoped mechanism, not a global one.

Co-Authored-By: Claude Code <noreply@anthropic.com>

src/controller/meson.build

@@ -232,6 +232,19 @@ controller_unit_test_deps = [
   ssl_dep,
 ]
 
+# bug-924: unit tests create /tmp/xce_* scratch dirs via mkdtemp() and never
+# remove them; a full run leaks 70+ dirs and fills /tmp to ENOSPC (seen at
+# 16238 dirs / 696G / disk 100%), surfacing as unrelated "Expected Non-NULL"
+# cascade failures. Link the mkdtemp auto-clean shim (testsuite_support/
+# mkdtemp_clean.c) into every executable that shares this deps list
+# (controller_unit_test + the standalone *_unit_test binaries below) so each
+# scratch dir is removed at process exit -- fixing the leak at the source
+# instead of masking it with xce-build.sh's age-gated prune.
+mkdtemp_clean_lib = static_library(
+    'xce_mkdtemp_clean',
+    join_paths(meson.source_root(), 'testsuite_support', 'mkdtemp_clean.c'))
+controller_unit_test_deps += declare_dependency(link_with: mkdtemp_clean_lib)
+
 controller_unit_test = executable('controller_unit_test',
                                   controller_unit_test_src,
                                   include_directories: [incdir],

src/scheduler/testsuite/meson.build

@@ -14,6 +14,18 @@ scheduler_test_src = [
 scheduler_test_deps = [libxce_deps, librtti_dep, unity_dep, libxce_dep,
                        libedit_dep]
 
+# bug-924: unit tests create /tmp/xce_* scratch dirs via mkdtemp() and never
+# remove them; a full run leaks dozens of dirs and fills /tmp to ENOSPC,
+# surfacing as unrelated "Expected Non-NULL"-style cascade failures. Link the
+# mkdtemp auto-clean shim (testsuite_support/mkdtemp_clean.c) into every
+# scheduler test executable that shares this deps list so each scratch dir is
+# removed at process exit -- fixing the leak at the source instead of masking
+# it with xce-build.sh's age-gated prune.
+mkdtemp_clean_lib = static_library(
+    'xce_mkdtemp_clean_sched',
+    join_paths(meson.source_root(), 'testsuite_support', 'mkdtemp_clean.c'))
+scheduler_test_deps += declare_dependency(link_with: mkdtemp_clean_lib)
+
 # unity.pc's Cflags only adds -I${includedir}/unity, so <unity.h> resolves
 # but <unity/unity.h> (used by every suite) does not — add the parent
 # includedir as well (same fix as src/lib/meson.build).

src/testsuite_support/mkdtemp_clean.c

@@ -0,0 +1,177 @@
+/*
+ * bug-924: unit tests create scratch dirs by calling mkdtemp() (almost all
+ * under /tmp/xce_*) and none of them clean up. A full ASAN run leaks 70+
+ * such dirs per invocation; left unchecked it fills the sandbox disk to
+ * ENOSPC, which then surfaces as unrelated "Expected Non-NULL"-style cascade
+ * failures in whatever test next calls mkdtemp()/persist_init() (seen at
+ * 16238 dirs / 696G / disk 100%). xce-build.sh's age-gated
+ * prune_stale_tmp_fixtures() only sweeps the debris hours later -- a mask,
+ * not a fix.
+ *
+ * This file is the fix at the source. It provides a replacement `mkdtemp()`
+ * that is linked into every unit-test executable in this project (via a
+ * declare_dependency() appended to each testsuite's shared *_test_deps list,
+ * see controller/meson.build and scheduler/testsuite/meson.build). Every
+ * scratch dir it creates is registered and recursively removed by an
+ * atexit() handler, so a test binary leaves /tmp clean regardless of which
+ * test "owns" the dir. Cleanup runs at process exit (after ALL TEST_
+ * functions), so tests that legitimately inspect a fixture's on-disk
+ * contents across multiple TEST functions in the same file are unaffected --
+ * the very property that made retrofitting cleanup into 70+ individual test
+ * files invasive.
+ *
+ * Contract is glibc mkdtemp's: `tmpl` must end in XXXXXX (overwritten with a
+ * unique directory name); returns `tmpl` on success, NULL on error. If the
+ * dir was subsequently removed by the test itself, the atexit sweep is a
+ * harmless no-op (rmdir fails with ENOENT).
+ */
+#define _GNU_SOURCE
+#include <dirent.h>
+#include <errno.h>
+#include <pthread.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <time.h>
+#include <unistd.h>
+
+#define MKDTEMP_SUFFIX "XXXXXX"
+#define MKDTEMP_TRIES  1000
+
+static pthread_mutex_t g_clean_mu       = PTHREAD_MUTEX_INITIALIZER;
+static char**          g_clean_dirs     = NULL;
+static size_t          g_clean_count    = 0;
+static size_t          g_clean_capacity = 0;
+static int             g_clean_armed    = 0;
+
+/* Recursively remove path (file or directory tree). Missing path is a
+ * no-op; symlinks are unlinked, never followed. */
+static void rmrf(const char* path)
+{
+    struct dirent* de;
+    DIR*           d;
+    char           child[4096];
+    struct stat    st;
+
+    d = opendir(path);
+    if (d == NULL) {
+        rmdir(path);
+        return;
+    }
+    while ((de = readdir(d)) != NULL) {
+        if (strcmp(de->d_name, ".") == 0 || strcmp(de->d_name, "..") == 0) {
+            continue;
+        }
+        if (snprintf(child, sizeof(child), "%s/%s", path, de->d_name)
+            >= (int)sizeof(child)) {
+            continue;
+        }
+        if (lstat(child, &st) == 0 && S_ISDIR(st.st_mode)) {
+            rmrf(child);
+        } else {
+            unlink(child);
+        }
+    }
+    closedir(d);
+    rmdir(path);
+}
+
+static void mkdtemp_cleanup_all(void)
+{
+    size_t i;
+
+    pthread_mutex_lock(&g_clean_mu);
+    for (i = 0; i < g_clean_count; i++) {
+        if (g_clean_dirs[i] != NULL) {
+            rmrf(g_clean_dirs[i]);
+            free(g_clean_dirs[i]);
+            g_clean_dirs[i] = NULL;
+        }
+    }
+    g_clean_count = 0;
+    pthread_mutex_unlock(&g_clean_mu);
+}
+
+/* Own a copy of `path` (the caller's tmpl buffer is reused across mkdtemp
+ * calls) and arrange for it to be removed at process exit (atexit). */
+static void track_dir(const char* path)
+{
+    char*  copy;
+    size_t ncap;
+    char** ndirs;
+
+    copy = strdup(path);
+    if (copy == NULL) {
+        return;
+    }
+    pthread_mutex_lock(&g_clean_mu);
+    if (!g_clean_armed) {
+        atexit(mkdtemp_cleanup_all);
+        g_clean_armed = 1;
+    }
+    if (g_clean_count == g_clean_capacity) {
+        ncap  = g_clean_capacity ? g_clean_capacity * 2 : 64;
+        ndirs = (char**)realloc(g_clean_dirs, ncap * sizeof(*ndirs));
+        if (ndirs == NULL) {
+            pthread_mutex_unlock(&g_clean_mu);
+            free(copy);
+            return;
+        }
+        g_clean_dirs     = ndirs;
+        g_clean_capacity = ncap;
+    }
+    g_clean_dirs[g_clean_count++] = copy;
+    pthread_mutex_unlock(&g_clean_mu);
+}
+
+static char mkdtemp_rand_char(void)
+{
+    static const char alphabet[] =
+        "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
+
+    return alphabet[rand() % (sizeof(alphabet) - 1)];
+}
+
+char* mkdtemp(char* tmpl)
+{
+    size_t len;
+    char*  suffix;
+    int    tries;
+    int    i;
+
+    /* glibc declares mkdtemp with __attribute__((nonnull(1))); tmpl is never
+     * NULL at a call site (it is the caller's writable buffer). */
+    len = strlen(tmpl);
+    if (len < 6 || strcmp(tmpl + len - 6, MKDTEMP_SUFFIX) != 0) {
+        errno = EINVAL;
+        return NULL;
+    }
+
+    /* Best-effort seed; uniqueness is guaranteed by the kernel-EEXIST retry
+     * loop below, not by the RNG. */
+    {
+        static int seeded = 0;
+        if (!seeded) {
+            srand((unsigned)(time(NULL) ^ (uintptr_t)&suffix));
+            seeded = 1;
+        }
+    }
+
+    suffix = tmpl + len - 6;
+    for (tries = 0; tries < MKDTEMP_TRIES; tries++) {
+        for (i = 0; i < 6; i++) {
+            suffix[i] = mkdtemp_rand_char();
+        }
+        if (mkdir(tmpl, 0700) == 0) {
+            track_dir(tmpl);
+            return tmpl;
+        }
+        if (errno != EEXIST) {
+            return NULL;
+        }
+    }
+    errno = EEXIST;
+    return NULL;
+}