---
id: "bug-1969"
status: "fixed"
status_updated: "2026-10-10"
status_note: ""
component: "autoscaler"
severity: "hang"
timestamp: "2026-10-10T13:29:02Z"
last_seen: "2026-10-10T13:29:02Z"
occurrences: 1
files: ["src/autoscaler/scaler/plug_mgr.c"]
tags: ["autoscaler", "fork", "threads", "deadlock"]
related: []
fix_commits: ["19052674"]
gitea_issue: 400
gitea_synced_status: "fixed"
gitea_synced_hash: "0a2b050e1505"
---

# bug-1969: autoscaler forks from a multi-threaded process and runs unsafe code in the child (plug_mgr.c)

## Description
**Observed:** (source reading, 2026-10-10; found while introducing the no-fork rule, `.kiro/steering/tech.md` "Process Creation")
The autoscaler is multi-threaded (`scaler.c:589`/`:596` start `scaler_monitor` and `scaler_demand_handler` with `pthread_create`). `plug_mgr.c` runs a provider script with `fork()` and, in the child, calls functions that take locks or allocate before `execl()`:
```c
pid = fork();                                   /* plug_mgr.c:441 */
...
} else if (pid == 0) {
    fd = open(out_file, ...);
    if (fd == -1) { ERROR("Create %s failed", out_file); exit(EXIT_FAILURE); }
    my_snprintf(cmd, sizeof(cmd), "%s/scripts/%s", ...);
    ...
    export_env(prov);                           /* setenv(): malloc */
    ret = execl("/bin/sh", "sh", cmd, "-f", in_file, (char*)NULL);
    if (ret == -1) { ERROR("execl() %s failed", cmd); exit(EXIT_FAILURE); }
```
If another thread holds the malloc arena or logger lock at the moment of `fork()`, the child deadlocks and the parent's `waitpid()` never returns; `exit()` also runs atexit handlers and flushes stdio copied from the parent.

**Expected:** the autoscaler starts provider scripts with `posix_spawn()` (output redirect via file actions, environment passed as `envp`), or forks only with async-signal-safe code before `exec` and `_exit()` on failure, so the no-fork rule can cover `src/autoscaler`.
**Impact:** hang: an intermittent deadlocked provider-script child blocks the scaling thread in `waitpid()`. The autoscaler is excluded from the no-fork rule only because it is not finalized; this must be fixed before it ships.
**Found by:** audit for the no-fork rule, 2026-10-10.

## Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: glibc 2.42-47 (by source reading; no binary run)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (xce.build)
- Code: scheduler-framework @ c4fee107-dirty
- Deployment: autoscaler with any provider plugin (found by source reading)

## Reproduction
1. Run the autoscaler with a provider while its monitor and demand threads are logging/allocating.
2. Trigger provider actions repeatedly (scale up/down).
3. Occasionally a provider child never reaches `execl()` and the autoscaler thread blocks in `waitpid()` (stack: child in `__lll_lock_wait` under `malloc`/logger).
- Frequency: deterministic (code path: the unsafe child code runs on every provider call; the deadlock itself is timing-dependent, not yet observed)
- Evidence: the code quoted in Description; no logs kept.

## Root cause
`invoke()` (`src/autoscaler/scaler/plug_mgr.c`) started each provider script with `fork()` + `execl()` from the multi-threaded scaler (`scaler.c:589`/`:596` start the monitor and demand threads), and the child called non-async-signal-safe functions (`ERROR()` logging, `my_snprintf`, `setenv()` in `export_env()`, `exit()`) before exec. A lock held by another thread at fork time (malloc arena, logger, environment lock) stays locked in the child forever. Pre-existing since the autoscaler port; never caught because its unit suite was not built.

## Fix
`plug_mgr.c`: new `run_script()` starts `sh <scriptPath>/scripts/<action> -f <in.json>` with `posix_spawn()`:
- stdout and stderr go to the output file through `posix_spawn_file_actions_addopen()`/`adddup2()`;
- the provider variables (`PRO_CONF_DIR`, `PRO_DATA_DIR`, `PRO_LOG_DIR`, `PROVIDER_NAME`) are passed as the child's `envp`, built by new `provider_env()` from the scaler's environment (replacing `export_env()`, which `setenv()`'d in the child);
- the child gets an empty signal mask and default dispositions (`POSIX_SPAWN_SETSIGMASK|SETSIGDEF`);
- `waitpid()` retries on `EINTR`; a spawn failure is logged and the call returns no output.
`src/autoscaler/meson.build`: the never-built scaler unit suite is now the meson test `autoscaler test` (runs under `/xce-build all`), linked with `-Wl,--wrap=fork`. The no-fork rule's scope is unchanged (autoscaler still excluded as not finalized); `tech.md` and `check-no-fork.py` now say its provider calls use `posix_spawn()`.

## Verification
- Test: `test_provider_call_does_not_fork` (`src/autoscaler/scaler/testsuite/test_plug_mgr.c`; `__wrap_fork` counts calls): RED `test.c:188:test_provider_call_does_not_fork:FAIL: Expected 0 Was 1. a provider call must not fork() (bug-1969)`, GREEN after. `test_provider_call_passes_environment` (new `testsuite/envscripts/scripts/getAvailableTemplates.sh` echoes the provider variables into the template id) guards that the script still sees `PROVIDER_NAME`/`PRO_CONF_DIR`/`PRO_DATA_DIR` and that the scaler's own environment is untouched; it passed before and after. The six existing plug_mgr tests pass: 12 Tests 0 Failures.
- Suites: `xce-build.sh all asan xce.build --incremental`, 2026-10-10: Ok 160 / Fail 0 (incl. new `autoscaler test` and `no-fork rule`), Go tests ok
- Live: n/a: the autoscaler library is linked by no shipped binary; verified with the fake provider scripts only

## References
- Fix commit: `19052674`
- Rule: `.kiro/steering/tech.md` "Process Creation" (no fork in multi-threaded processes)
- Superseding work: `.kiro/specs/resource-provisioner/` (may replace this code)