docs(buglog): log worker signal and launch-type issues (bug-1954..bug-1956)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
8568c8ab80
commit
6445367c47
GPG key ID:
80533536B0EB651C
4 changed files with 278 additions and 0 deletions
|
|
@ -18232,6 +18232,81 @@
|
|||
],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1953.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1954",
|
||||
"status": "open",
|
||||
"status_updated": "2026-10-10",
|
||||
"status_note": "newly logged",
|
||||
"component": "worker",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-10-10T08:46:07Z",
|
||||
"last_seen": "2026-10-10T08:46:07Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "xcecd_worker sends SIGTERM for a KILL request without force and SIGSTOP for a TSTP request\n\n**Observed:** (source reading, 2026-10-10; found while fixing bug-1940)\n`worker_signal_callback()` (`src/worker/src/xcecd_worker.c:1115-1130`)\nreplaces the requested signal:\n```c\ncase SIGTERM:\ncase SIGKILL:\n if (force) { success = signal_gid_except_self(SIGKILL); }\n else { success = signal_gid_except_self(SIGTERM); }\n...\ncase SIGSTOP:\ncase SIGTSTP:\n success = signal_gid_except_self(SIGSTOP);\n```\nSince bug-1940 the requested signal reaches the worker, but an explicit\n`kill -s KILL` without force still sends SIGTERM, and an explicit TSTP is sent\nas the uncatchable SIGSTOP.\n\n**Expected:** the job receives the signal the user asked for; force kill\n(SIGKILL) stays available for a default kill.\n**Impact:** wrong result: a job that ignores SIGTERM survives an explicit\nSIGKILL request; a job that handles SIGTSTP to checkpoint or clean up cannot\ndo so.\n**Found by:** review of the bug-1940/1942 fixes, 2026-10-10.",
|
||||
"file": "src/worker/src/xcecd_worker.c",
|
||||
"root_cause": "TODO: required for fixed/partial",
|
||||
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
|
||||
"tags": [
|
||||
"worker",
|
||||
"job-control",
|
||||
"signal"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1940"
|
||||
],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1954.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1955",
|
||||
"status": "open",
|
||||
"status_updated": "2026-10-10",
|
||||
"status_note": "newly logged",
|
||||
"component": "worker",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-10-10T08:46:10Z",
|
||||
"last_seen": "2026-10-10T08:46:10Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "xcecd drops a signal for a remote task of a multi-host job that is still starting\n\n**Observed:** (source reading, 2026-10-10; found while fixing bug-1942)\nOnly the head host's job start opens the pending-signal window\n(`pending_signal_start(d->jobid)`, `src/worker/src/controller_connector.c:1062`,\nbug-1908). The runtasks handler for remote tasks (`handle_runtasks()`, `:1147`)\nnever does, so a signal that arrives before the remote task's worker entry is\nregistered is logged and dropped:\n```c\nXCECD_LOG(XCE_LOG_ERROR, \"Failed to find worker for job ID: %s\", jobidStr);\nFREEUP(task_ctx);\nreturn -1; /* controller_connector.c:1606 */\n```\nSince bug-1942 every host of a multi-host job receives job control, so this\nwindow is now reachable.\n\n**Expected:** like the head host (bug-1908), a signal for a remote task that\nis starting is kept and delivered once the task runs.\n**Impact:** wrong result: a kill or stop sent right after a multi-host job\nstarts can miss some of its remote tasks, which keep running.\n**Found by:** review of the bug-1940/1942 fixes, 2026-10-10.",
|
||||
"file": "src/worker/src/controller_connector.c",
|
||||
"root_cause": "TODO: required for fixed/partial",
|
||||
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
|
||||
"tags": [
|
||||
"worker",
|
||||
"job-control",
|
||||
"parallel",
|
||||
"signal"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1942",
|
||||
"bug-1908"
|
||||
],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1955.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1956",
|
||||
"status": "investigating",
|
||||
"status_updated": "2026-10-10",
|
||||
"status_note": "suspected: confirm whether a job without launch type decodes to NULL at the worker",
|
||||
"component": "worker",
|
||||
"severity": "crash",
|
||||
"timestamp": "2026-10-10T08:46:13Z",
|
||||
"last_seen": "2026-10-10T08:46:13Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "xcecd compares a NULL launch_type with strcmp when a job has no launch type (possible crash)\n\n**Observed:** (source reading, 2026-10-10; suspected, not reproduced)\n`job_start_local_task()` stores NULL when the start info has no launch type\nand then dereferences it:\n```c\nif (d->start_info->launch_type) {\n job_ctx->startinfo->launch_type = strdup(...);\n} else {\n job_ctx->startinfo->launch_type = NULL; /* :981-985 */\n}\n...\nif (strcmp(job_ctx->startinfo->launch_type, \"openmpi\") == 0 /* :994 */\n```\n`job_start_remote_task()` does the same without a check (`:753`). The apiserver\nsends no launch type when the user gives none (`cStringOrNil`,\n`backend/convert.go:339`). Jobs without a launch type do run in existing\ntests, so the decoder may deliver an empty string rather than NULL: not\nconfirmed either way.\n\n**Expected:** a missing launch type means a plain (non-MPI) job; no NULL\ndereference.\n**Impact:** possible worker crash at job start for jobs without a launch\ntype.\n**Found by:** review of the bug-1940/1942 fixes, 2026-10-10.",
|
||||
"file": "src/worker/src/controller_connector.c",
|
||||
"root_cause": "TODO: required for fixed/partial",
|
||||
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
|
||||
"tags": [
|
||||
"worker",
|
||||
"crash",
|
||||
"launch-type"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1956.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
69
.wolf/bugs/bug-1954.md
Normal file
69
.wolf/bugs/bug-1954.md
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
---
|
||||
id: "bug-1954"
|
||||
status: "open"
|
||||
status_updated: "2026-10-10"
|
||||
status_note: "newly logged"
|
||||
component: "worker"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-10T08:46:07Z"
|
||||
last_seen: "2026-10-10T08:46:07Z"
|
||||
occurrences: 1
|
||||
files: ["src/worker/src/xcecd_worker.c"]
|
||||
tags: ["worker", "job-control", "signal"]
|
||||
related: ["bug-1940"]
|
||||
---
|
||||
|
||||
# bug-1954: xcecd_worker sends SIGTERM for a KILL request without force and SIGSTOP for a TSTP request
|
||||
|
||||
## Description
|
||||
**Observed:** (source reading, 2026-10-10; found while fixing bug-1940)
|
||||
`worker_signal_callback()` (`src/worker/src/xcecd_worker.c:1115-1130`)
|
||||
replaces the requested signal:
|
||||
```c
|
||||
case SIGTERM:
|
||||
case SIGKILL:
|
||||
if (force) { success = signal_gid_except_self(SIGKILL); }
|
||||
else { success = signal_gid_except_self(SIGTERM); }
|
||||
...
|
||||
case SIGSTOP:
|
||||
case SIGTSTP:
|
||||
success = signal_gid_except_self(SIGSTOP);
|
||||
```
|
||||
Since bug-1940 the requested signal reaches the worker, but an explicit
|
||||
`kill -s KILL` without force still sends SIGTERM, and an explicit TSTP is sent
|
||||
as the uncatchable SIGSTOP.
|
||||
|
||||
**Expected:** the job receives the signal the user asked for; force kill
|
||||
(SIGKILL) stays available for a default kill.
|
||||
**Impact:** wrong result: a job that ignores SIGTERM survives an explicit
|
||||
SIGKILL request; a job that handles SIGTSTP to checkpoint or clean up cannot
|
||||
do so.
|
||||
**Found by:** review of the bug-1940/1942 fixes, 2026-10-10.
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (xce.build)
|
||||
- Code: scheduler-framework @ 8568c8ab-dirty
|
||||
- Deployment: real `xcecd` worker (systemd/dbus); multi-host for the remote-task case
|
||||
|
||||
## Reproduction
|
||||
1. Run a job that traps TERM (`trap '' TERM; sleep 600`).
|
||||
2. `POST /xce/v1/jobs/<id>/control` with action kill, signal `SIGKILL`,
|
||||
force_kill false.
|
||||
3. The job keeps running (it received SIGTERM).
|
||||
- Frequency: deterministic by code reading (not yet run)
|
||||
- Evidence: the code quoted in Description; no logs kept.
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
65
.wolf/bugs/bug-1955.md
Normal file
65
.wolf/bugs/bug-1955.md
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
---
|
||||
id: "bug-1955"
|
||||
status: "open"
|
||||
status_updated: "2026-10-10"
|
||||
status_note: "newly logged"
|
||||
component: "worker"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-10T08:46:10Z"
|
||||
last_seen: "2026-10-10T08:46:10Z"
|
||||
occurrences: 1
|
||||
files: ["src/worker/src/controller_connector.c"]
|
||||
tags: ["worker", "job-control", "parallel", "signal"]
|
||||
related: ["bug-1942", "bug-1908"]
|
||||
---
|
||||
|
||||
# bug-1955: xcecd drops a signal for a remote task of a multi-host job that is still starting
|
||||
|
||||
## Description
|
||||
**Observed:** (source reading, 2026-10-10; found while fixing bug-1942)
|
||||
Only the head host's job start opens the pending-signal window
|
||||
(`pending_signal_start(d->jobid)`, `src/worker/src/controller_connector.c:1062`,
|
||||
bug-1908). The runtasks handler for remote tasks (`handle_runtasks()`, `:1147`)
|
||||
never does, so a signal that arrives before the remote task's worker entry is
|
||||
registered is logged and dropped:
|
||||
```c
|
||||
XCECD_LOG(XCE_LOG_ERROR, "Failed to find worker for job ID: %s", jobidStr);
|
||||
FREEUP(task_ctx);
|
||||
return -1; /* controller_connector.c:1606 */
|
||||
```
|
||||
Since bug-1942 every host of a multi-host job receives job control, so this
|
||||
window is now reachable.
|
||||
|
||||
**Expected:** like the head host (bug-1908), a signal for a remote task that
|
||||
is starting is kept and delivered once the task runs.
|
||||
**Impact:** wrong result: a kill or stop sent right after a multi-host job
|
||||
starts can miss some of its remote tasks, which keep running.
|
||||
**Found by:** review of the bug-1940/1942 fixes, 2026-10-10.
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (xce.build)
|
||||
- Code: scheduler-framework @ 8568c8ab-dirty
|
||||
- Deployment: real `xcecd` worker (systemd/dbus); multi-host for the remote-task case
|
||||
|
||||
## Reproduction
|
||||
1. Submit a multi-host job (e.g. `/xce-test-multihost` topology) and kill it
|
||||
immediately after dispatch.
|
||||
2. A remote host logs "Failed to find worker for job ID" and its task keeps
|
||||
running.
|
||||
- Frequency: deterministic by code reading (not yet run)
|
||||
- Evidence: the code quoted in Description; no logs kept.
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
69
.wolf/bugs/bug-1956.md
Normal file
69
.wolf/bugs/bug-1956.md
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
---
|
||||
id: "bug-1956"
|
||||
status: "investigating"
|
||||
status_updated: "2026-10-10"
|
||||
status_note: "suspected: confirm whether a job without launch type decodes to NULL at the worker"
|
||||
component: "worker"
|
||||
severity: "crash"
|
||||
timestamp: "2026-10-10T08:46:13Z"
|
||||
last_seen: "2026-10-10T08:46:13Z"
|
||||
occurrences: 1
|
||||
files: ["src/worker/src/controller_connector.c"]
|
||||
tags: ["worker", "crash", "launch-type"]
|
||||
related: []
|
||||
---
|
||||
|
||||
# bug-1956: xcecd compares a NULL launch_type with strcmp when a job has no launch type (possible crash)
|
||||
|
||||
## Description
|
||||
**Observed:** (source reading, 2026-10-10; suspected, not reproduced)
|
||||
`job_start_local_task()` stores NULL when the start info has no launch type
|
||||
and then dereferences it:
|
||||
```c
|
||||
if (d->start_info->launch_type) {
|
||||
job_ctx->startinfo->launch_type = strdup(...);
|
||||
} else {
|
||||
job_ctx->startinfo->launch_type = NULL; /* :981-985 */
|
||||
}
|
||||
...
|
||||
if (strcmp(job_ctx->startinfo->launch_type, "openmpi") == 0 /* :994 */
|
||||
```
|
||||
`job_start_remote_task()` does the same without a check (`:753`). The apiserver
|
||||
sends no launch type when the user gives none (`cStringOrNil`,
|
||||
`backend/convert.go:339`). Jobs without a launch type do run in existing
|
||||
tests, so the decoder may deliver an empty string rather than NULL: not
|
||||
confirmed either way.
|
||||
|
||||
**Expected:** a missing launch type means a plain (non-MPI) job; no NULL
|
||||
dereference.
|
||||
**Impact:** possible worker crash at job start for jobs without a launch
|
||||
type.
|
||||
**Found by:** review of the bug-1940/1942 fixes, 2026-10-10.
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (xce.build)
|
||||
- Code: scheduler-framework @ 8568c8ab-dirty
|
||||
- Deployment: real `xcecd` worker (systemd/dbus); multi-host for the remote-task case
|
||||
|
||||
## Reproduction
|
||||
1. Decode a job start message whose start info has no launch type and run
|
||||
`job_start_local_task()` (worker unit test), or submit a job without a
|
||||
launch type through libxce/harness to a real `xcecd`.
|
||||
2. Check whether `startinfo->launch_type` is NULL at `:994`.
|
||||
- Frequency: deterministic by code reading (not yet run)
|
||||
- Evidence: the code quoted in Description; no logs kept.
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
Loading…
Add table
Add a link
Reference in a new issue