From 6445367c4749fa9d418ea87f661171c8a8006a51 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 10 Oct 2026 08:46:24 +0000 Subject: [PATCH] docs(buglog): log worker signal and launch-type issues (bug-1954..bug-1956) Co-Authored-By: Claude Opus 5.5 (1M context) --- .wolf/buglog.json | 75 ++++++++++++++++++++++++++++++++++++++++++ .wolf/bugs/bug-1954.md | 69 ++++++++++++++++++++++++++++++++++++++ .wolf/bugs/bug-1955.md | 65 ++++++++++++++++++++++++++++++++++++ .wolf/bugs/bug-1956.md | 69 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 278 insertions(+) create mode 100644 .wolf/bugs/bug-1954.md create mode 100644 .wolf/bugs/bug-1955.md create mode 100644 .wolf/bugs/bug-1956.md diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 54380cc3..e4bec0d5 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -18232,6 +18232,81 @@ ], "fix_commits": [], "record": ".wolf/bugs/bug-1953.md" + }, + { + "id": "bug-1954", + "status": "open", + "status_updated": "2026-10-10", + "status_note": "newly logged", + "component": "worker", + "severity": "wrong-result", + "timestamp": "2026-10-10T08:46:07Z", + "last_seen": "2026-10-10T08:46:07Z", + "occurrences": 1, + "error_message": "xcecd_worker sends SIGTERM for a KILL request without force and SIGSTOP for a TSTP request\n\n**Observed:** (source reading, 2026-10-10; found while fixing bug-1940)\n`worker_signal_callback()` (`src/worker/src/xcecd_worker.c:1115-1130`)\nreplaces the requested signal:\n```c\ncase SIGTERM:\ncase SIGKILL:\n if (force) { success = signal_gid_except_self(SIGKILL); }\n else { success = signal_gid_except_self(SIGTERM); }\n...\ncase SIGSTOP:\ncase SIGTSTP:\n success = signal_gid_except_self(SIGSTOP);\n```\nSince bug-1940 the requested signal reaches the worker, but an explicit\n`kill -s KILL` without force still sends SIGTERM, and an explicit TSTP is sent\nas the uncatchable SIGSTOP.\n\n**Expected:** the job receives the signal the user asked for; force kill\n(SIGKILL) stays available for a default kill.\n**Impact:** wrong result: a job that ignores SIGTERM survives an explicit\nSIGKILL request; a job that handles SIGTSTP to checkpoint or clean up cannot\ndo so.\n**Found by:** review of the bug-1940/1942 fixes, 2026-10-10.", + "file": "src/worker/src/xcecd_worker.c", + "root_cause": "TODO: required for fixed/partial", + "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "tags": [ + "worker", + "job-control", + "signal" + ], + "related_bugs": [ + "bug-1940" + ], + "fix_commits": [], + "record": ".wolf/bugs/bug-1954.md" + }, + { + "id": "bug-1955", + "status": "open", + "status_updated": "2026-10-10", + "status_note": "newly logged", + "component": "worker", + "severity": "wrong-result", + "timestamp": "2026-10-10T08:46:10Z", + "last_seen": "2026-10-10T08:46:10Z", + "occurrences": 1, + "error_message": "xcecd drops a signal for a remote task of a multi-host job that is still starting\n\n**Observed:** (source reading, 2026-10-10; found while fixing bug-1942)\nOnly the head host's job start opens the pending-signal window\n(`pending_signal_start(d->jobid)`, `src/worker/src/controller_connector.c:1062`,\nbug-1908). The runtasks handler for remote tasks (`handle_runtasks()`, `:1147`)\nnever does, so a signal that arrives before the remote task's worker entry is\nregistered is logged and dropped:\n```c\nXCECD_LOG(XCE_LOG_ERROR, \"Failed to find worker for job ID: %s\", jobidStr);\nFREEUP(task_ctx);\nreturn -1; /* controller_connector.c:1606 */\n```\nSince bug-1942 every host of a multi-host job receives job control, so this\nwindow is now reachable.\n\n**Expected:** like the head host (bug-1908), a signal for a remote task that\nis starting is kept and delivered once the task runs.\n**Impact:** wrong result: a kill or stop sent right after a multi-host job\nstarts can miss some of its remote tasks, which keep running.\n**Found by:** review of the bug-1940/1942 fixes, 2026-10-10.", + "file": "src/worker/src/controller_connector.c", + "root_cause": "TODO: required for fixed/partial", + "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "tags": [ + "worker", + "job-control", + "parallel", + "signal" + ], + "related_bugs": [ + "bug-1942", + "bug-1908" + ], + "fix_commits": [], + "record": ".wolf/bugs/bug-1955.md" + }, + { + "id": "bug-1956", + "status": "investigating", + "status_updated": "2026-10-10", + "status_note": "suspected: confirm whether a job without launch type decodes to NULL at the worker", + "component": "worker", + "severity": "crash", + "timestamp": "2026-10-10T08:46:13Z", + "last_seen": "2026-10-10T08:46:13Z", + "occurrences": 1, + "error_message": "xcecd compares a NULL launch_type with strcmp when a job has no launch type (possible crash)\n\n**Observed:** (source reading, 2026-10-10; suspected, not reproduced)\n`job_start_local_task()` stores NULL when the start info has no launch type\nand then dereferences it:\n```c\nif (d->start_info->launch_type) {\n job_ctx->startinfo->launch_type = strdup(...);\n} else {\n job_ctx->startinfo->launch_type = NULL; /* :981-985 */\n}\n...\nif (strcmp(job_ctx->startinfo->launch_type, \"openmpi\") == 0 /* :994 */\n```\n`job_start_remote_task()` does the same without a check (`:753`). The apiserver\nsends no launch type when the user gives none (`cStringOrNil`,\n`backend/convert.go:339`). Jobs without a launch type do run in existing\ntests, so the decoder may deliver an empty string rather than NULL: not\nconfirmed either way.\n\n**Expected:** a missing launch type means a plain (non-MPI) job; no NULL\ndereference.\n**Impact:** possible worker crash at job start for jobs without a launch\ntype.\n**Found by:** review of the bug-1940/1942 fixes, 2026-10-10.", + "file": "src/worker/src/controller_connector.c", + "root_cause": "TODO: required for fixed/partial", + "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "tags": [ + "worker", + "crash", + "launch-type" + ], + "related_bugs": [], + "fix_commits": [], + "record": ".wolf/bugs/bug-1956.md" } ] } diff --git a/.wolf/bugs/bug-1954.md b/.wolf/bugs/bug-1954.md new file mode 100644 index 00000000..1d71faa2 --- /dev/null +++ b/.wolf/bugs/bug-1954.md @@ -0,0 +1,69 @@ +--- +id: "bug-1954" +status: "open" +status_updated: "2026-10-10" +status_note: "newly logged" +component: "worker" +severity: "wrong-result" +timestamp: "2026-10-10T08:46:07Z" +last_seen: "2026-10-10T08:46:07Z" +occurrences: 1 +files: ["src/worker/src/xcecd_worker.c"] +tags: ["worker", "job-control", "signal"] +related: ["bug-1940"] +--- + +# bug-1954: xcecd_worker sends SIGTERM for a KILL request without force and SIGSTOP for a TSTP request + +## Description +**Observed:** (source reading, 2026-10-10; found while fixing bug-1940) +`worker_signal_callback()` (`src/worker/src/xcecd_worker.c:1115-1130`) +replaces the requested signal: +```c +case SIGTERM: +case SIGKILL: + if (force) { success = signal_gid_except_self(SIGKILL); } + else { success = signal_gid_except_self(SIGTERM); } +... +case SIGSTOP: +case SIGTSTP: + success = signal_gid_except_self(SIGSTOP); +``` +Since bug-1940 the requested signal reaches the worker, but an explicit +`kill -s KILL` without force still sends SIGTERM, and an explicit TSTP is sent +as the uncatchable SIGSTOP. + +**Expected:** the job receives the signal the user asked for; force kill +(SIGKILL) stays available for a default kill. +**Impact:** wrong result: a job that ignores SIGTERM survives an explicit +SIGKILL request; a job that handles SIGTSTP to checkpoint or clean up cannot +do so. +**Found by:** review of the bug-1940/1942 fixes, 2026-10-10. +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (xce.build) +- Code: scheduler-framework @ 8568c8ab-dirty +- Deployment: real `xcecd` worker (systemd/dbus); multi-host for the remote-task case + +## Reproduction +1. Run a job that traps TERM (`trap '' TERM; sleep 600`). +2. `POST /xce/v1/jobs//control` with action kill, signal `SIGKILL`, + force_kill false. +3. The job keeps running (it received SIGTERM). +- Frequency: deterministic by code reading (not yet run) +- Evidence: the code quoted in Description; no logs kept. +## Root cause +TODO: required for fixed/partial + +## Fix +TODO: required for fixed/partial + +## Verification +- Test: TODO: name + file; RED before the fix, GREEN after +- Suites: TODO: suite results and build profile +- Live: TODO: soak/deploy run, or n/a and why + +## References +- Related: none diff --git a/.wolf/bugs/bug-1955.md b/.wolf/bugs/bug-1955.md new file mode 100644 index 00000000..b278eec7 --- /dev/null +++ b/.wolf/bugs/bug-1955.md @@ -0,0 +1,65 @@ +--- +id: "bug-1955" +status: "open" +status_updated: "2026-10-10" +status_note: "newly logged" +component: "worker" +severity: "wrong-result" +timestamp: "2026-10-10T08:46:10Z" +last_seen: "2026-10-10T08:46:10Z" +occurrences: 1 +files: ["src/worker/src/controller_connector.c"] +tags: ["worker", "job-control", "parallel", "signal"] +related: ["bug-1942", "bug-1908"] +--- + +# bug-1955: xcecd drops a signal for a remote task of a multi-host job that is still starting + +## Description +**Observed:** (source reading, 2026-10-10; found while fixing bug-1942) +Only the head host's job start opens the pending-signal window +(`pending_signal_start(d->jobid)`, `src/worker/src/controller_connector.c:1062`, +bug-1908). The runtasks handler for remote tasks (`handle_runtasks()`, `:1147`) +never does, so a signal that arrives before the remote task's worker entry is +registered is logged and dropped: +```c +XCECD_LOG(XCE_LOG_ERROR, "Failed to find worker for job ID: %s", jobidStr); +FREEUP(task_ctx); +return -1; /* controller_connector.c:1606 */ +``` +Since bug-1942 every host of a multi-host job receives job control, so this +window is now reachable. + +**Expected:** like the head host (bug-1908), a signal for a remote task that +is starting is kept and delivered once the task runs. +**Impact:** wrong result: a kill or stop sent right after a multi-host job +starts can miss some of its remote tasks, which keep running. +**Found by:** review of the bug-1940/1942 fixes, 2026-10-10. +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (xce.build) +- Code: scheduler-framework @ 8568c8ab-dirty +- Deployment: real `xcecd` worker (systemd/dbus); multi-host for the remote-task case + +## Reproduction +1. Submit a multi-host job (e.g. `/xce-test-multihost` topology) and kill it + immediately after dispatch. +2. A remote host logs "Failed to find worker for job ID" and its task keeps + running. +- Frequency: deterministic by code reading (not yet run) +- Evidence: the code quoted in Description; no logs kept. +## Root cause +TODO: required for fixed/partial + +## Fix +TODO: required for fixed/partial + +## Verification +- Test: TODO: name + file; RED before the fix, GREEN after +- Suites: TODO: suite results and build profile +- Live: TODO: soak/deploy run, or n/a and why + +## References +- Related: none diff --git a/.wolf/bugs/bug-1956.md b/.wolf/bugs/bug-1956.md new file mode 100644 index 00000000..d76d1fee --- /dev/null +++ b/.wolf/bugs/bug-1956.md @@ -0,0 +1,69 @@ +--- +id: "bug-1956" +status: "investigating" +status_updated: "2026-10-10" +status_note: "suspected: confirm whether a job without launch type decodes to NULL at the worker" +component: "worker" +severity: "crash" +timestamp: "2026-10-10T08:46:13Z" +last_seen: "2026-10-10T08:46:13Z" +occurrences: 1 +files: ["src/worker/src/controller_connector.c"] +tags: ["worker", "crash", "launch-type"] +related: [] +--- + +# bug-1956: xcecd compares a NULL launch_type with strcmp when a job has no launch type (possible crash) + +## Description +**Observed:** (source reading, 2026-10-10; suspected, not reproduced) +`job_start_local_task()` stores NULL when the start info has no launch type +and then dereferences it: +```c +if (d->start_info->launch_type) { + job_ctx->startinfo->launch_type = strdup(...); +} else { + job_ctx->startinfo->launch_type = NULL; /* :981-985 */ +} +... +if (strcmp(job_ctx->startinfo->launch_type, "openmpi") == 0 /* :994 */ +``` +`job_start_remote_task()` does the same without a check (`:753`). The apiserver +sends no launch type when the user gives none (`cStringOrNil`, +`backend/convert.go:339`). Jobs without a launch type do run in existing +tests, so the decoder may deliver an empty string rather than NULL: not +confirmed either way. + +**Expected:** a missing launch type means a plain (non-MPI) job; no NULL +dereference. +**Impact:** possible worker crash at job start for jobs without a launch +type. +**Found by:** review of the bug-1940/1942 fixes, 2026-10-10. +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (xce.build) +- Code: scheduler-framework @ 8568c8ab-dirty +- Deployment: real `xcecd` worker (systemd/dbus); multi-host for the remote-task case + +## Reproduction +1. Decode a job start message whose start info has no launch type and run + `job_start_local_task()` (worker unit test), or submit a job without a + launch type through libxce/harness to a real `xcecd`. +2. Check whether `startinfo->launch_type` is NULL at `:994`. +- Frequency: deterministic by code reading (not yet run) +- Evidence: the code quoted in Description; no logs kept. +## Root cause +TODO: required for fixed/partial + +## Fix +TODO: required for fixed/partial + +## Verification +- Test: TODO: name + file; RED before the fix, GREEN after +- Suites: TODO: suite results and build profile +- Live: TODO: soak/deploy run, or n/a and why + +## References +- Related: none