myxce/.wolf/bugs/bug-1969.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1969	fixed	2026-10-10		autoscaler	hang	2026-10-10T13:29:02Z	2026-10-10T13:29:02Z	1	src/autoscaler/scaler/plug_mgr.c	autoscaler fork threads deadlock		19052674	400	fixed	0a2b050e1505

bug-1969: autoscaler forks from a multi-threaded process and runs unsafe code in the child (plug_mgr.c)

Description

Observed: (source reading, 2026-10-10; found while introducing the no-fork rule, .kiro/steering/tech.md "Process Creation") The autoscaler is multi-threaded (scaler.c:589/:596 start scaler_monitor and scaler_demand_handler with pthread_create). plug_mgr.c runs a provider script with fork() and, in the child, calls functions that take locks or allocate before execl():

pid = fork();                                   /* plug_mgr.c:441 */
...
} else if (pid == 0) {
    fd = open(out_file, ...);
    if (fd == -1) { ERROR("Create %s failed", out_file); exit(EXIT_FAILURE); }
    my_snprintf(cmd, sizeof(cmd), "%s/scripts/%s", ...);
    ...
    export_env(prov);                           /* setenv(): malloc */
    ret = execl("/bin/sh", "sh", cmd, "-f", in_file, (char*)NULL);
    if (ret == -1) { ERROR("execl() %s failed", cmd); exit(EXIT_FAILURE); }

If another thread holds the malloc arena or logger lock at the moment of fork(), the child deadlocks and the parent's waitpid() never returns; exit() also runs atexit handlers and flushes stdio copied from the parent.

Expected: the autoscaler starts provider scripts with posix_spawn() (output redirect via file actions, environment passed as envp), or forks only with async-signal-safe code before exec and _exit() on failure, so the no-fork rule can cover src/autoscaler. Impact: hang: an intermittent deadlocked provider-script child blocks the scaling thread in waitpid(). The autoscaler is excluded from the no-fork rule only because it is not finalized; this must be fixed before it ships. Found by: audit for the no-fork rule, 2026-10-10.

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (by source reading; no binary run)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ c4fee107-dirty
• Deployment: autoscaler with any provider plugin (found by source reading)

Reproduction

1. Run the autoscaler with a provider while its monitor and demand threads are logging/allocating.
2. Trigger provider actions repeatedly (scale up/down).
3. Occasionally a provider child never reaches execl() and the autoscaler thread blocks in waitpid() (stack: child in __lll_lock_wait under malloc/logger).

• Frequency: deterministic (code path: the unsafe child code runs on every provider call; the deadlock itself is timing-dependent, not yet observed)
• Evidence: the code quoted in Description; no logs kept.

Root cause

invoke() (src/autoscaler/scaler/plug_mgr.c) started each provider script with fork() + execl() from the multi-threaded scaler (scaler.c:589/:596 start the monitor and demand threads), and the child called non-async-signal-safe functions (ERROR() logging, my_snprintf, setenv() in export_env(), exit()) before exec. A lock held by another thread at fork time (malloc arena, logger, environment lock) stays locked in the child forever. Pre-existing since the autoscaler port; never caught because its unit suite was not built.

Fix

plug_mgr.c: new run_script() starts sh <scriptPath>/scripts/<action> -f <in.json> with posix_spawn():

• stdout and stderr go to the output file through posix_spawn_file_actions_addopen()/adddup2();
• the provider variables (PRO_CONF_DIR, PRO_DATA_DIR, PRO_LOG_DIR, PROVIDER_NAME) are passed as the child's envp, built by new provider_env() from the scaler's environment (replacing export_env(), which setenv()'d in the child);
• the child gets an empty signal mask and default dispositions (POSIX_SPAWN_SETSIGMASK|SETSIGDEF);
• waitpid() retries on EINTR; a spawn failure is logged and the call returns no output. src/autoscaler/meson.build: the never-built scaler unit suite is now the meson test autoscaler test (runs under /xce-build all), linked with -Wl,--wrap=fork. The no-fork rule's scope is unchanged (autoscaler still excluded as not finalized); tech.md and check-no-fork.py now say its provider calls use posix_spawn().

Verification

• Test: test_provider_call_does_not_fork (src/autoscaler/scaler/testsuite/test_plug_mgr.c; __wrap_fork counts calls): RED test.c:188:test_provider_call_does_not_fork:FAIL: Expected 0 Was 1. a provider call must not fork() (bug-1969), GREEN after. test_provider_call_passes_environment (new testsuite/envscripts/scripts/getAvailableTemplates.sh echoes the provider variables into the template id) guards that the script still sees PROVIDER_NAME/PRO_CONF_DIR/PRO_DATA_DIR and that the scaler's own environment is untouched; it passed before and after. The six existing plug_mgr tests pass: 12 Tests 0 Failures.
• Suites: xce-build.sh all asan xce.build --incremental, 2026-10-10: Ok 160 / Fail 0 (incl. new autoscaler test and no-fork rule), Go tests ok
• Live: n/a: the autoscaler library is linked by no shipped binary; verified with the fake provider scripts only

References

• Fix commit: 19052674
• Rule: .kiro/steering/tech.md "Process Creation" (no fork in multi-threaded processes)
• Superseding work: .kiro/specs/resource-provisioner/ (may replace this code)