docs(buglog): log bug-1969 (autoscaler forks from a multi-threaded process)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
6dd70f1091
commit
832e397351
2 changed files with 90 additions and 0 deletions
|
|
@ -18735,6 +18735,30 @@
|
|||
"record": ".wolf/bugs/bug-1968.md",
|
||||
"gitea_issue": 399,
|
||||
"gitea_synced_status": "open"
|
||||
},
|
||||
{
|
||||
"id": "bug-1969",
|
||||
"status": "open",
|
||||
"status_updated": "2026-10-10",
|
||||
"status_note": "newly logged",
|
||||
"component": "autoscaler",
|
||||
"severity": "hang",
|
||||
"timestamp": "2026-10-10T13:29:02Z",
|
||||
"last_seen": "2026-10-10T13:29:02Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "autoscaler forks from a multi-threaded process and runs unsafe code in the child (plug_mgr.c)\n\n**Observed:** (source reading, 2026-10-10; found while introducing the no-fork rule, `.kiro/steering/tech.md` \"Process Creation\")\nThe autoscaler is multi-threaded (`scaler.c:589`/`:596` start `scaler_monitor` and `scaler_demand_handler` with `pthread_create`). `plug_mgr.c` runs a provider script with `fork()` and, in the child, calls functions that take locks or allocate before `execl()`:\n```c\npid = fork(); /* plug_mgr.c:441 */\n...\n} else if (pid == 0) {\n fd = open(out_file, ...);\n if (fd == -1) { ERROR(\"Create %s failed\", out_file); exit(EXIT_FAILURE); }\n my_snprintf(cmd, sizeof(cmd), \"%s/scripts/%s\", ...);\n ...\n export_env(prov); /* setenv(): malloc */\n ret = execl(\"/bin/sh\", \"sh\", cmd, \"-f\", in_file, (char*)NULL);\n if (ret == -1) { ERROR(\"execl() %s failed\", cmd); exit(EXIT_FAILURE); }\n```\nIf another thread holds the malloc arena or logger lock at the moment of `fork()`, the child deadlocks and the parent's `waitpid()` never returns; `exit()` also runs atexit handlers and flushes stdio copied from the parent.\n\n**Expected:** the autoscaler starts provider scripts with `posix_spawn()` (output redirect via file actions, environment passed as `envp`), or forks only with async-signal-safe code before `exec` and `_exit()` on failure, so the no-fork rule can cover `src/autoscaler`.\n**Impact:** hang: an intermittent deadlocked provider-script child blocks the scaling thread in `waitpid()`. The autoscaler is excluded from the no-fork rule only because it is not finalized; this must be fixed before it ships.\n**Found by:** audit for the no-fork rule, 2026-10-10.",
|
||||
"file": "src/autoscaler/scaler/plug_mgr.c",
|
||||
"root_cause": "TODO: required for fixed/partial",
|
||||
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
|
||||
"tags": [
|
||||
"autoscaler",
|
||||
"fork",
|
||||
"threads",
|
||||
"deadlock"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1969.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
66
.wolf/bugs/bug-1969.md
Normal file
66
.wolf/bugs/bug-1969.md
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
---
|
||||
id: "bug-1969"
|
||||
status: "open"
|
||||
status_updated: "2026-10-10"
|
||||
status_note: "newly logged"
|
||||
component: "autoscaler"
|
||||
severity: "hang"
|
||||
timestamp: "2026-10-10T13:29:02Z"
|
||||
last_seen: "2026-10-10T13:29:02Z"
|
||||
occurrences: 1
|
||||
files: ["src/autoscaler/scaler/plug_mgr.c"]
|
||||
tags: ["autoscaler", "fork", "threads", "deadlock"]
|
||||
related: []
|
||||
---
|
||||
|
||||
# bug-1969: autoscaler forks from a multi-threaded process and runs unsafe code in the child (plug_mgr.c)
|
||||
|
||||
## Description
|
||||
**Observed:** (source reading, 2026-10-10; found while introducing the no-fork rule, `.kiro/steering/tech.md` "Process Creation")
|
||||
The autoscaler is multi-threaded (`scaler.c:589`/`:596` start `scaler_monitor` and `scaler_demand_handler` with `pthread_create`). `plug_mgr.c` runs a provider script with `fork()` and, in the child, calls functions that take locks or allocate before `execl()`:
|
||||
```c
|
||||
pid = fork(); /* plug_mgr.c:441 */
|
||||
...
|
||||
} else if (pid == 0) {
|
||||
fd = open(out_file, ...);
|
||||
if (fd == -1) { ERROR("Create %s failed", out_file); exit(EXIT_FAILURE); }
|
||||
my_snprintf(cmd, sizeof(cmd), "%s/scripts/%s", ...);
|
||||
...
|
||||
export_env(prov); /* setenv(): malloc */
|
||||
ret = execl("/bin/sh", "sh", cmd, "-f", in_file, (char*)NULL);
|
||||
if (ret == -1) { ERROR("execl() %s failed", cmd); exit(EXIT_FAILURE); }
|
||||
```
|
||||
If another thread holds the malloc arena or logger lock at the moment of `fork()`, the child deadlocks and the parent's `waitpid()` never returns; `exit()` also runs atexit handlers and flushes stdio copied from the parent.
|
||||
|
||||
**Expected:** the autoscaler starts provider scripts with `posix_spawn()` (output redirect via file actions, environment passed as `envp`), or forks only with async-signal-safe code before `exec` and `_exit()` on failure, so the no-fork rule can cover `src/autoscaler`.
|
||||
**Impact:** hang: an intermittent deadlocked provider-script child blocks the scaling thread in `waitpid()`. The autoscaler is excluded from the no-fork rule only because it is not finalized; this must be fixed before it ships.
|
||||
**Found by:** audit for the no-fork rule, 2026-10-10.
|
||||
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (by source reading; no binary run)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (xce.build)
|
||||
- Code: scheduler-framework @ c4fee107-dirty
|
||||
- Deployment: autoscaler with any provider plugin (found by source reading)
|
||||
|
||||
## Reproduction
|
||||
1. Run the autoscaler with a provider while its monitor and demand threads are logging/allocating.
|
||||
2. Trigger provider actions repeatedly (scale up/down).
|
||||
3. Occasionally a provider child never reaches `execl()` and the autoscaler thread blocks in `waitpid()` (stack: child in `__lll_lock_wait` under `malloc`/logger).
|
||||
- Frequency: deterministic (code path: the unsafe child code runs on every provider call; the deadlock itself is timing-dependent, not yet observed)
|
||||
- Evidence: the code quoted in Description; no logs kept.
|
||||
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
Loading…
Add table
Add a link
Reference in a new issue