myxce/.wolf/bugs/bug-1908.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits
bug-1908	fixed	2026-10-02		worker	wrong-result	2026-10-02T02:27:01Z	2026-10-02T02:27:01Z	1	src/worker/src/controller_connector.c	job-control kill race dispatch	bug-1905	8b7173bb

bug-1908: a kill sent right after dispatch is lost: the worker drops it before registering the job

Description

Observed:

05,850 controller_connector.c:563  Job ID: 1-0-1790907905-32768[2], Command: sleep 600 ...   (dispatch received)
07,318 controller_connector.c:1546 Received job control request to send SIGINFO for job ID: 1-0-1790907905-32768[2]
07,318 controller_connector.c:1586 Failed to find worker for job ID: 1-0-1790907905-32768[2]
09,690 xcecd_worker_utils.c:352    Job ID 1-0-1790907905-32768[2] running - sending status to controller

xcecli jobs list already showed the job RUN, xcecli jobs kill answered job <...> is being <kill> (apiserver overallStatus accepted), but the worker had not yet added the job to its worker list, dropped the request, and the job then ran to completion. Same for a plain job (1-0-1790907905-0: dispatch 06,876, kill 07,313 dropped, running 09,724).

Expected: a kill accepted for a job the controller shows as RUN takes effect: the worker applies it once the job is registered (or the controller retries / reports the kill as not delivered). Impact: wrong result: a kill issued in the first seconds after dispatch is reported as accepted but silently lost; the job keeps running. The lookup branch is shared by the real worker (xcecd), not only xcecd_virtual (real worker not tested). Found by: test (live verification of bug-1905, 2026-10-01)

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (src/build.jai)
• Code: scheduler-framework @ 8ac637ad-dirty
• Deployment: 3-node HA (xce-deploy-ha) + scheduler + apiserver + one xcecd_virtual worker, single host

Reproduction

1. Deploy with xce-deploy-ha + start-all.sh; xcecli login -u admin -p admin --url http://localhost:8080.
2. xcecli jobs submit -J plain sleep 600 </dev/null; as soon as xcecli jobs list shows RUN (about 1 s), xcecli jobs kill <id> </dev/null.
3. The worker log shows Failed to find worker for job ID, and the job stays RUN.

• Frequency: 2/2 kills sent within ~1.5 s of dispatch, 2026-10-01
• Evidence: /home/claude/tmp/kill-race/worker.log

Root cause

The controller forwards a job-control signal as soon as the job has an allocation (src/controller/job.c job-control handler -> xce_jobctrl_req). The worker answers the start request by queuing it on its thread pool (controller_connector.c start handler -> submit_fork_task) and only adds the job to its worker list when xcecd_process_command runs (xcecd.c), which for the first jobs after a worker starts was ~3 s later; the real worker can also only signal a job once the job's IPC client has connected. handle_jobcontrol (controller_connector.c) dropped a signal for a job not in the list ("Failed to find worker", -1) and also for a listed job still PENDING/STARTING, so the kill was lost while the API had already reported it accepted.

Fix

New src/worker/src/pending_signal.c (include/pending_signal.h): a thread-safe table of jobs whose start the worker has accepted. The start handler calls pending_signal_start() before queuing the start; handle_jobcontrol keeps a signal for a queued or PENDING/STARTING job with pending_signal_record() and replies success; update_worker_status() takes it on the STARTING -> RUNNING transition and delivers it (virtual: worker->killed_signal; real: ipc_master_send_job_signal, whose client is connected once the job reports RUNNING). xcecd_job_start_error() and the DONE/EXITED transitions drop the entry. A signal for a job the worker knows nothing about still fails as before.

Verification

• Test: test_jobcontrol_signal_for_queued_start_is_kept (RED before the fix: Expected 0 Was -1) and test_jobcontrol_signal_for_unknown_job_fails in src/worker/testsuite/test_controller_connector.c; table semantics in the new src/worker/testsuite/test_pending_signal.c (6 tests); GREEN after
• Suites: worker meson suite 15/15 OK (asan, src/build.jai)
• Live: fresh xce-deploy-ha + xcecli, kills sent as soon as the jobs listed RUN: the worker logged has not started yet; signal 15 will be delivered when it runs for plain and arr[2] 2.3 s before they started, then delivering signal 15 kept while it was starting; both ended EXIT while arr[1]/arr[3] kept running; a later kill of the running arr[1] still worked (/home/claude/tmp/bug-1908/fixed-run.log). The real worker's IPC delivery path is not exercised here (no systemd/dbus in this container).

References

• Related: none