lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.wolf/bugs/bug-1905.md
Rongsong Shen 7d6a6e6043
docs(buglog): bug-1905 fixed in 8ac637ad; log bug-1908 (kill lost before worker registers job) 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-02 02:27:34 +00:00

4.3 KiB
Raw Blame History

id status status_updated status_note component severity timestamp last_seen occurrences files tags related fix_commits
bug-1905 fixed 2026-10-02 worker wrong-result 2026-10-01T14:32:18Z 2026-10-01T14:32:18Z 1
src/worker/src/controller_connector.c
src/worker/src/xcecd_dbus_manager.c
virtual-cluster
job-control
kill
8ac637ad

bug-1905: xcecd_virtual ignores kill of a running job: job keeps RUN until its sleep time elapses

Description

Observed:

14:17:47,625 controller_connector.c:1546 Received job control request to send SIGINFO for job ID: 1-0-1790864259-0
14:17:47,625 controller_connector.c:1610 Job control task handled directly for job ID: 1-0-1790864259-0
14:27:40,008 xcecd_worker_utils.c:847  Sleep time 600 >= 600 seconds elapsed for worker 1009 (job 1-0-1790864259-0)
ctrl-1: on_job_finish job finish jobid 1-0-1790864259-0 exit code 0

xcecli jobs kill of a RUN job (plain sleep 600, and array elements arr3[2], arr3[3]) is accepted (apiserver: overallStatus accepted, SIGTERM) and reaches the virtual worker, but the job stays RUN and finishes DONE with exit 0 when its sleep time elapses. Killing a PEND job works (EXIT).

Expected: the virtual worker ends a killed running job as EXITED with exit code 128+signal on its next poll (the killed_signal branch in xcecd_dbus_manager.c:259-262). Impact: kill of running jobs cannot be exercised on virtual-worker deployments (tests, soaks); the real worker path (ipc_master_send_job_signal) is not affected by this record. Found by: spec task (job-array-one-based-index task 10.2 live REST flow)

Environment

  • OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
  • libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
  • Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
  • Build: asan (src/build.jai)
  • Code: scheduler-framework @ 6439e585-dirty
  • Deployment: 3-node HA (xce-deploy-ha) + scheduler + apiserver + one xcecd_virtual worker, single host

Reproduction

  1. Deploy with xce-deploy-ha and start-all.sh; xcecli login -u admin -p admin --url http://localhost:8080.
  2. xcecli jobs submit -J plain2 sleep 600 </dev/null; wait until xcecli jobs list shows RUN.
  3. xcecli jobs kill <id> </dev/null; xcecli jobs list -a <id> still shows RUN, and the job ends DONE after 600 s.
  • Frequency: deterministic (3/3 kills of running jobs, 2026-10-01)
  • Evidence: /home/claude/tmp/bug-1905/kill-trace.log

Root cause

dbus_workers_update() (src/worker/src/xcecd_dbus_manager.c) decided a virtual job's end inside the branches of get_systemd_job_exit_code(): the == 0 branch checked the sleep timer and then killed_signal, but the < 0 branch checked only the timer. A virtual job has no systemd unit, and without a system bus (this container: Failed to connect to system bus) the call returns -1, so the kill recorded by the job-control handler (controller_connector.c, worker->killed_signal) was never acted on; the job stayed RUN and ended DONE with exit 0 when its sleep time elapsed. (Even on the == 0 path the timer was checked before the kill.)

Fix

In VIRTUAL_CLUSTER builds the loop decides a running job's end before asking systemd: killed_signal != 0 ends it as WORKER_EXITED with exit code 128 + signal; otherwise it is WORKER_DONE once its sleep time is up. The two partial virtual copies inside the exit-code branches were removed; real-worker behaviour is unchanged.

Verification

  • Test: test_killed_running_job_exits_without_systemd (and the guard test_unkilled_running_job_keeps_running) in src/worker/testsuite/test_dbus_workers_update.c; RED before the fix (no status update, Expected 1 Was 0), GREEN after
  • Suites: worker meson suite 14/14 OK (asan, src/build.jai)
  • Live: xce-deploy-ha + xcecli, kill sent after the worker logged the jobs running: plain2 and arr2[2] went RUN -> EXIT within 6 s, arr2[1]/arr2[3] stayed RUN (/home/claude/tmp/bug-1905/fixed-run.log). A kill sent within ~a few seconds of dispatch, before the worker has registered the job, is still lost ("Failed to find worker") -- a separate defect shared with the real worker, see References (bug-1908)

References

  • Related: bug-1908 (kill lost when sent before the worker registers the job)