docs(buglog): bug-1905 fixed in 8ac637ad; log bug-1908 (kill lost before worker registers job)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-02 02:27:34 +00:00

3 KiB

Raw Blame History

status

status_updated

status_note

component

severity

timestamp

last_seen

occurrences

files

bug-1908: a kill sent right after dispatch is lost: the worker drops it before registering the job

Description

Observed:

05,850 controller_connector.c:563  Job ID: 1-0-1790907905-32768[2], Command: sleep 600 ...   (dispatch received)
07,318 controller_connector.c:1546 Received job control request to send SIGINFO for job ID: 1-0-1790907905-32768[2]
07,318 controller_connector.c:1586 Failed to find worker for job ID: 1-0-1790907905-32768[2]
09,690 xcecd_worker_utils.c:352    Job ID 1-0-1790907905-32768[2] running - sending status to controller

xcecli jobs list already showed the job RUN, xcecli jobs kill answered job <...> is being <kill> (apiserver overallStatus accepted), but the worker had not yet added the job to its worker list, dropped the request, and the job then ran to completion. Same for a plain job (1-0-1790907905-0: dispatch 06,876, kill 07,313 dropped, running 09,724).

Expected: a kill accepted for a job the controller shows as RUN takes effect: the worker applies it once the job is registered (or the controller retries / reports the kill as not delivered). Impact: wrong result: a kill issued in the first seconds after dispatch is reported as accepted but silently lost; the job keeps running. The lookup branch is shared by the real worker (xcecd), not only xcecd_virtual (real worker not tested). Found by: test (live verification of bug-1905, 2026-10-01)

Environment

OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
Build: asan (src/build.jai)
Code: scheduler-framework @ 8ac637ad-dirty
Deployment: 3-node HA (xce-deploy-ha) + scheduler + apiserver + one xcecd_virtual worker, single host

Reproduction

Deploy with xce-deploy-ha + start-all.sh; xcecli login -u admin -p admin --url http://localhost:8080.
xcecli jobs submit -J plain sleep 600 </dev/null; as soon as xcecli jobs list shows RUN (about 1 s), xcecli jobs kill <id> </dev/null.
The worker log shows Failed to find worker for job ID, and the job stays RUN.

Frequency: 2/2 kills sent within ~1.5 s of dispatch, 2026-10-01
Evidence: /home/claude/tmp/kill-race/worker.log

Root cause

TODO: required for fixed/partial

Fix