From 832e3973516760f103ac0d716a1d193f4a7fbd0a Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 10 Oct 2026 13:36:02 +0000 Subject: [PATCH] docs(buglog): log bug-1969 (autoscaler forks from a multi-threaded process) Co-Authored-By: Claude Opus 5.5 (1M context) --- .wolf/buglog.json | 24 +++++++++++++++ .wolf/bugs/bug-1969.md | 66 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+) create mode 100644 .wolf/bugs/bug-1969.md diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 1f4f4a1b..d90b608a 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -18735,6 +18735,30 @@ "record": ".wolf/bugs/bug-1968.md", "gitea_issue": 399, "gitea_synced_status": "open" + }, + { + "id": "bug-1969", + "status": "open", + "status_updated": "2026-10-10", + "status_note": "newly logged", + "component": "autoscaler", + "severity": "hang", + "timestamp": "2026-10-10T13:29:02Z", + "last_seen": "2026-10-10T13:29:02Z", + "occurrences": 1, + "error_message": "autoscaler forks from a multi-threaded process and runs unsafe code in the child (plug_mgr.c)\n\n**Observed:** (source reading, 2026-10-10; found while introducing the no-fork rule, `.kiro/steering/tech.md` \"Process Creation\")\nThe autoscaler is multi-threaded (`scaler.c:589`/`:596` start `scaler_monitor` and `scaler_demand_handler` with `pthread_create`). `plug_mgr.c` runs a provider script with `fork()` and, in the child, calls functions that take locks or allocate before `execl()`:\n```c\npid = fork(); /* plug_mgr.c:441 */\n...\n} else if (pid == 0) {\n fd = open(out_file, ...);\n if (fd == -1) { ERROR(\"Create %s failed\", out_file); exit(EXIT_FAILURE); }\n my_snprintf(cmd, sizeof(cmd), \"%s/scripts/%s\", ...);\n ...\n export_env(prov); /* setenv(): malloc */\n ret = execl(\"/bin/sh\", \"sh\", cmd, \"-f\", in_file, (char*)NULL);\n if (ret == -1) { ERROR(\"execl() %s failed\", cmd); exit(EXIT_FAILURE); }\n```\nIf another thread holds the malloc arena or logger lock at the moment of `fork()`, the child deadlocks and the parent's `waitpid()` never returns; `exit()` also runs atexit handlers and flushes stdio copied from the parent.\n\n**Expected:** the autoscaler starts provider scripts with `posix_spawn()` (output redirect via file actions, environment passed as `envp`), or forks only with async-signal-safe code before `exec` and `_exit()` on failure, so the no-fork rule can cover `src/autoscaler`.\n**Impact:** hang: an intermittent deadlocked provider-script child blocks the scaling thread in `waitpid()`. The autoscaler is excluded from the no-fork rule only because it is not finalized; this must be fixed before it ships.\n**Found by:** audit for the no-fork rule, 2026-10-10.", + "file": "src/autoscaler/scaler/plug_mgr.c", + "root_cause": "TODO: required for fixed/partial", + "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "tags": [ + "autoscaler", + "fork", + "threads", + "deadlock" + ], + "related_bugs": [], + "fix_commits": [], + "record": ".wolf/bugs/bug-1969.md" } ] } diff --git a/.wolf/bugs/bug-1969.md b/.wolf/bugs/bug-1969.md new file mode 100644 index 00000000..cba6a45b --- /dev/null +++ b/.wolf/bugs/bug-1969.md @@ -0,0 +1,66 @@ +--- +id: "bug-1969" +status: "open" +status_updated: "2026-10-10" +status_note: "newly logged" +component: "autoscaler" +severity: "hang" +timestamp: "2026-10-10T13:29:02Z" +last_seen: "2026-10-10T13:29:02Z" +occurrences: 1 +files: ["src/autoscaler/scaler/plug_mgr.c"] +tags: ["autoscaler", "fork", "threads", "deadlock"] +related: [] +--- + +# bug-1969: autoscaler forks from a multi-threaded process and runs unsafe code in the child (plug_mgr.c) + +## Description +**Observed:** (source reading, 2026-10-10; found while introducing the no-fork rule, `.kiro/steering/tech.md` "Process Creation") +The autoscaler is multi-threaded (`scaler.c:589`/`:596` start `scaler_monitor` and `scaler_demand_handler` with `pthread_create`). `plug_mgr.c` runs a provider script with `fork()` and, in the child, calls functions that take locks or allocate before `execl()`: +```c +pid = fork(); /* plug_mgr.c:441 */ +... +} else if (pid == 0) { + fd = open(out_file, ...); + if (fd == -1) { ERROR("Create %s failed", out_file); exit(EXIT_FAILURE); } + my_snprintf(cmd, sizeof(cmd), "%s/scripts/%s", ...); + ... + export_env(prov); /* setenv(): malloc */ + ret = execl("/bin/sh", "sh", cmd, "-f", in_file, (char*)NULL); + if (ret == -1) { ERROR("execl() %s failed", cmd); exit(EXIT_FAILURE); } +``` +If another thread holds the malloc arena or logger lock at the moment of `fork()`, the child deadlocks and the parent's `waitpid()` never returns; `exit()` also runs atexit handlers and flushes stdio copied from the parent. + +**Expected:** the autoscaler starts provider scripts with `posix_spawn()` (output redirect via file actions, environment passed as `envp`), or forks only with async-signal-safe code before `exec` and `_exit()` on failure, so the no-fork rule can cover `src/autoscaler`. +**Impact:** hang: an intermittent deadlocked provider-script child blocks the scaling thread in `waitpid()`. The autoscaler is excluded from the no-fork rule only because it is not finalized; this must be fixed before it ships. +**Found by:** audit for the no-fork rule, 2026-10-10. + +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (by source reading; no binary run) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (xce.build) +- Code: scheduler-framework @ c4fee107-dirty +- Deployment: autoscaler with any provider plugin (found by source reading) + +## Reproduction +1. Run the autoscaler with a provider while its monitor and demand threads are logging/allocating. +2. Trigger provider actions repeatedly (scale up/down). +3. Occasionally a provider child never reaches `execl()` and the autoscaler thread blocks in `waitpid()` (stack: child in `__lll_lock_wait` under `malloc`/logger). +- Frequency: deterministic (code path: the unsafe child code runs on every provider call; the deadlock itself is timing-dependent, not yet observed) +- Evidence: the code quoted in Description; no logs kept. + +## Root cause +TODO: required for fixed/partial + +## Fix +TODO: required for fixed/partial + +## Verification +- Test: TODO: name + file; RED before the fix, GREEN after +- Suites: TODO: suite results and build profile +- Live: TODO: soak/deploy run, or n/a and why + +## References +- Related: none