diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 7ebfcaa5..ab6ebb18 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -18740,9 +18740,9 @@ }, { "id": "bug-1969", - "status": "open", + "status": "fixed", "status_updated": "2026-10-10", - "status_note": "newly logged", + "status_note": "", "component": "autoscaler", "severity": "hang", "timestamp": "2026-10-10T13:29:02Z", @@ -18750,8 +18750,8 @@ "occurrences": 1, "error_message": "autoscaler forks from a multi-threaded process and runs unsafe code in the child (plug_mgr.c)\n\n**Observed:** (source reading, 2026-10-10; found while introducing the no-fork rule, `.kiro/steering/tech.md` \"Process Creation\")\nThe autoscaler is multi-threaded (`scaler.c:589`/`:596` start `scaler_monitor` and `scaler_demand_handler` with `pthread_create`). `plug_mgr.c` runs a provider script with `fork()` and, in the child, calls functions that take locks or allocate before `execl()`:\n```c\npid = fork(); /* plug_mgr.c:441 */\n...\n} else if (pid == 0) {\n fd = open(out_file, ...);\n if (fd == -1) { ERROR(\"Create %s failed\", out_file); exit(EXIT_FAILURE); }\n my_snprintf(cmd, sizeof(cmd), \"%s/scripts/%s\", ...);\n ...\n export_env(prov); /* setenv(): malloc */\n ret = execl(\"/bin/sh\", \"sh\", cmd, \"-f\", in_file, (char*)NULL);\n if (ret == -1) { ERROR(\"execl() %s failed\", cmd); exit(EXIT_FAILURE); }\n```\nIf another thread holds the malloc arena or logger lock at the moment of `fork()`, the child deadlocks and the parent's `waitpid()` never returns; `exit()` also runs atexit handlers and flushes stdio copied from the parent.\n\n**Expected:** the autoscaler starts provider scripts with `posix_spawn()` (output redirect via file actions, environment passed as `envp`), or forks only with async-signal-safe code before `exec` and `_exit()` on failure, so the no-fork rule can cover `src/autoscaler`.\n**Impact:** hang: an intermittent deadlocked provider-script child blocks the scaling thread in `waitpid()`. The autoscaler is excluded from the no-fork rule only because it is not finalized; this must be fixed before it ships.\n**Found by:** audit for the no-fork rule, 2026-10-10.", "file": "src/autoscaler/scaler/plug_mgr.c", - "root_cause": "TODO: required for fixed/partial", - "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "root_cause": "`invoke()` (`src/autoscaler/scaler/plug_mgr.c`) started each provider script with `fork()` + `execl()` from the multi-threaded scaler (`scaler.c:589`/`:596` start the monitor and demand threads), and the child called non-async-signal-safe functions (`ERROR()` logging, `my_snprintf`, `setenv()` in `export_env()`, `exit()`) before exec. A lock held by another thread at fork time (malloc arena, logger, environment lock) stays locked in the child forever. Pre-existing since the autoscaler port; never caught because its unit suite was not built.", + "fix": "`plug_mgr.c`: new `run_script()` starts `sh /scripts/ -f ` with `posix_spawn()`:\n- stdout and stderr go to the output file through `posix_spawn_file_actions_addopen()`/`adddup2()`;\n- the provider variables (`PRO_CONF_DIR`, `PRO_DATA_DIR`, `PRO_LOG_DIR`, `PROVIDER_NAME`) are passed as the child's `envp`, built by new `provider_env()` from the scaler's environment (replacing `export_env()`, which `setenv()`'d in the child);\n- the child gets an empty signal mask and default dispositions (`POSIX_SPAWN_SETSIGMASK|SETSIGDEF`);\n- `waitpid()` retries on `EINTR`; a spawn failure is logged and the call returns no output.\n`src/autoscaler/meson.build`: the never-built scaler unit suite is now the meson test `autoscaler test` (runs under `/xce-build all`), linked with `-Wl,--wrap=fork`. The no-fork rule's scope is unchanged (autoscaler still excluded as not finalized); `tech.md` and `check-no-fork.py` now say its provider calls use `posix_spawn()`.\n\nVerification:\n- Test: `test_provider_call_does_not_fork` (`src/autoscaler/scaler/testsuite/test_plug_mgr.c`; `__wrap_fork` counts calls): RED `test.c:188:test_provider_call_does_not_fork:FAIL: Expected 0 Was 1. a provider call must not fork() (bug-1969)`, GREEN after. `test_provider_call_passes_environment` (new `testsuite/envscripts/scripts/getAvailableTemplates.sh` echoes the provider variables into the template id) guards that the script still sees `PROVIDER_NAME`/`PRO_CONF_DIR`/`PRO_DATA_DIR` and that the scaler's own environment is untouched; it passed before and after. The six existing plug_mgr tests pass: 12 Tests 0 Failures.\n- Suites: `xce-build.sh all asan xce.build --incremental`, 2026-10-10: Ok 160 / Fail 0 (incl. new `autoscaler test` and `no-fork rule`), Go tests ok\n- Live: n/a: the autoscaler library is linked by no shipped binary; verified with the fake provider scripts only", "tags": [ "autoscaler", "fork", @@ -18759,7 +18759,9 @@ "deadlock" ], "related_bugs": [], - "fix_commits": [], + "fix_commits": [ + "19052674" + ], "record": ".wolf/bugs/bug-1969.md", "gitea_issue": 400, "gitea_synced_status": "open" diff --git a/.wolf/bugs/bug-1969.md b/.wolf/bugs/bug-1969.md index 562cb531..3656097d 100644 --- a/.wolf/bugs/bug-1969.md +++ b/.wolf/bugs/bug-1969.md @@ -1,8 +1,8 @@ --- id: "bug-1969" -status: "open" +status: "fixed" status_updated: "2026-10-10" -status_note: "newly logged" +status_note: "" component: "autoscaler" severity: "hang" timestamp: "2026-10-10T13:29:02Z" @@ -11,6 +11,7 @@ occurrences: 1 files: ["src/autoscaler/scaler/plug_mgr.c"] tags: ["autoscaler", "fork", "threads", "deadlock"] related: [] +fix_commits: ["19052674"] gitea_issue: 400 gitea_synced_status: "open" gitea_synced_hash: "bcdb2d7efdf4" @@ -55,15 +56,23 @@ If another thread holds the malloc arena or logger lock at the moment of `fork() - Evidence: the code quoted in Description; no logs kept. ## Root cause -TODO: required for fixed/partial +`invoke()` (`src/autoscaler/scaler/plug_mgr.c`) started each provider script with `fork()` + `execl()` from the multi-threaded scaler (`scaler.c:589`/`:596` start the monitor and demand threads), and the child called non-async-signal-safe functions (`ERROR()` logging, `my_snprintf`, `setenv()` in `export_env()`, `exit()`) before exec. A lock held by another thread at fork time (malloc arena, logger, environment lock) stays locked in the child forever. Pre-existing since the autoscaler port; never caught because its unit suite was not built. ## Fix -TODO: required for fixed/partial +`plug_mgr.c`: new `run_script()` starts `sh /scripts/ -f ` with `posix_spawn()`: +- stdout and stderr go to the output file through `posix_spawn_file_actions_addopen()`/`adddup2()`; +- the provider variables (`PRO_CONF_DIR`, `PRO_DATA_DIR`, `PRO_LOG_DIR`, `PROVIDER_NAME`) are passed as the child's `envp`, built by new `provider_env()` from the scaler's environment (replacing `export_env()`, which `setenv()`'d in the child); +- the child gets an empty signal mask and default dispositions (`POSIX_SPAWN_SETSIGMASK|SETSIGDEF`); +- `waitpid()` retries on `EINTR`; a spawn failure is logged and the call returns no output. +`src/autoscaler/meson.build`: the never-built scaler unit suite is now the meson test `autoscaler test` (runs under `/xce-build all`), linked with `-Wl,--wrap=fork`. The no-fork rule's scope is unchanged (autoscaler still excluded as not finalized); `tech.md` and `check-no-fork.py` now say its provider calls use `posix_spawn()`. ## Verification -- Test: TODO: name + file; RED before the fix, GREEN after -- Suites: TODO: suite results and build profile -- Live: TODO: soak/deploy run, or n/a and why +- Test: `test_provider_call_does_not_fork` (`src/autoscaler/scaler/testsuite/test_plug_mgr.c`; `__wrap_fork` counts calls): RED `test.c:188:test_provider_call_does_not_fork:FAIL: Expected 0 Was 1. a provider call must not fork() (bug-1969)`, GREEN after. `test_provider_call_passes_environment` (new `testsuite/envscripts/scripts/getAvailableTemplates.sh` echoes the provider variables into the template id) guards that the script still sees `PROVIDER_NAME`/`PRO_CONF_DIR`/`PRO_DATA_DIR` and that the scaler's own environment is untouched; it passed before and after. The six existing plug_mgr tests pass: 12 Tests 0 Failures. +- Suites: `xce-build.sh all asan xce.build --incremental`, 2026-10-10: Ok 160 / Fail 0 (incl. new `autoscaler test` and `no-fork rule`), Go tests ok +- Live: n/a: the autoscaler library is linked by no shipped binary; verified with the fake provider scripts only ## References -- Related: none +- Fix commit: `19052674` +- Rule: `.kiro/steering/tech.md` "Process Creation" (no fork in multi-threaded processes) +- Superseding work: `.kiro/specs/resource-provisioner/` (may replace this code) +