60 lines
4.3 KiB
Markdown
60 lines
4.3 KiB
Markdown
---
|
|
id: "bug-1905"
|
|
status: "fixed"
|
|
status_updated: "2026-10-02"
|
|
status_note: ""
|
|
component: "worker"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-10-01T14:32:18Z"
|
|
last_seen: "2026-10-01T14:32:18Z"
|
|
occurrences: 1
|
|
files: ["src/worker/src/controller_connector.c", "src/worker/src/xcecd_dbus_manager.c"]
|
|
tags: ["virtual-cluster", "job-control", "kill"]
|
|
related: []
|
|
fix_commits: ["8ac637ad"]
|
|
---
|
|
|
|
# bug-1905: xcecd_virtual ignores kill of a running job: job keeps RUN until its sleep time elapses
|
|
|
|
## Description
|
|
**Observed:**
|
|
```
|
|
14:17:47,625 controller_connector.c:1546 Received job control request to send SIGINFO for job ID: 1-0-1790864259-0
|
|
14:17:47,625 controller_connector.c:1610 Job control task handled directly for job ID: 1-0-1790864259-0
|
|
14:27:40,008 xcecd_worker_utils.c:847 Sleep time 600 >= 600 seconds elapsed for worker 1009 (job 1-0-1790864259-0)
|
|
ctrl-1: on_job_finish job finish jobid 1-0-1790864259-0 exit code 0
|
|
```
|
|
`xcecli jobs kill` of a RUN job (plain `sleep 600`, and array elements `arr3[2]`, `arr3[3]`) is accepted (apiserver: `overallStatus accepted`, SIGTERM) and reaches the virtual worker, but the job stays RUN and finishes DONE with exit 0 when its sleep time elapses. Killing a PEND job works (EXIT).
|
|
|
|
**Expected:** the virtual worker ends a killed running job as EXITED with exit code 128+signal on its next poll (the `killed_signal` branch in `xcecd_dbus_manager.c:259-262`).
|
|
**Impact:** kill of running jobs cannot be exercised on virtual-worker deployments (tests, soaks); the real worker path (`ipc_master_send_job_signal`) is not affected by this record.
|
|
**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow)
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (src/build.jai)
|
|
- Code: scheduler-framework @ 6439e585-dirty
|
|
- Deployment: 3-node HA (xce-deploy-ha) + scheduler + apiserver + one xcecd_virtual worker, single host
|
|
|
|
## Reproduction
|
|
1. Deploy with xce-deploy-ha and `start-all.sh`; `xcecli login -u admin -p admin --url http://localhost:8080`.
|
|
2. `xcecli jobs submit -J plain2 sleep 600 </dev/null`; wait until `xcecli jobs list` shows RUN.
|
|
3. `xcecli jobs kill <id> </dev/null`; `xcecli jobs list -a <id>` still shows RUN, and the job ends DONE after 600 s.
|
|
- Frequency: deterministic (3/3 kills of running jobs, 2026-10-01)
|
|
- Evidence: /home/claude/tmp/bug-1905/kill-trace.log
|
|
|
|
## Root cause
|
|
`dbus_workers_update()` (`src/worker/src/xcecd_dbus_manager.c`) decided a virtual job's end inside the branches of `get_systemd_job_exit_code()`: the `== 0` branch checked the sleep timer and then `killed_signal`, but the `< 0` branch checked only the timer. A virtual job has no systemd unit, and without a system bus (this container: `Failed to connect to system bus`) the call returns -1, so the kill recorded by the job-control handler (`controller_connector.c`, `worker->killed_signal`) was never acted on; the job stayed RUN and ended DONE with exit 0 when its sleep time elapsed. (Even on the `== 0` path the timer was checked before the kill.)
|
|
|
|
## Fix
|
|
In `VIRTUAL_CLUSTER` builds the loop decides a running job's end before asking systemd: `killed_signal != 0` ends it as `WORKER_EXITED` with exit code 128 + signal; otherwise it is `WORKER_DONE` once its sleep time is up. The two partial virtual copies inside the exit-code branches were removed; real-worker behaviour is unchanged.
|
|
|
|
## Verification
|
|
- Test: `test_killed_running_job_exits_without_systemd` (and the guard `test_unkilled_running_job_keeps_running`) in `src/worker/testsuite/test_dbus_workers_update.c`; RED before the fix (no status update, `Expected 1 Was 0`), GREEN after
|
|
- Suites: worker meson suite 14/14 OK (asan, src/build.jai)
|
|
- Live: xce-deploy-ha + xcecli, kill sent after the worker logged the jobs running: `plain2` and `arr2[2]` went RUN -> EXIT within 6 s, `arr2[1]`/`arr2[3]` stayed RUN (/home/claude/tmp/bug-1905/fixed-run.log). A kill sent within ~a few seconds of dispatch, before the worker has registered the job, is still lost ("Failed to find worker") -- a separate defect shared with the real worker, see References (bug-1908)
|
|
|
|
## References
|
|
- Related: bug-1908 (kill lost when sent before the worker registers the job)
|