60 lines
5 KiB
Markdown
60 lines
5 KiB
Markdown
---
|
|
id: "bug-1908"
|
|
status: "fixed"
|
|
status_updated: "2026-10-02"
|
|
status_note: ""
|
|
component: "worker"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-10-02T02:27:01Z"
|
|
last_seen: "2026-10-02T02:27:01Z"
|
|
occurrences: 1
|
|
files: ["src/worker/src/controller_connector.c"]
|
|
tags: ["job-control", "kill", "race", "dispatch"]
|
|
related: ["bug-1905"]
|
|
fix_commits: ["8b7173bb"]
|
|
---
|
|
|
|
# bug-1908: a kill sent right after dispatch is lost: the worker drops it before registering the job
|
|
|
|
## Description
|
|
**Observed:**
|
|
```
|
|
05,850 controller_connector.c:563 Job ID: 1-0-1790907905-32768[2], Command: sleep 600 ... (dispatch received)
|
|
07,318 controller_connector.c:1546 Received job control request to send SIGINFO for job ID: 1-0-1790907905-32768[2]
|
|
07,318 controller_connector.c:1586 Failed to find worker for job ID: 1-0-1790907905-32768[2]
|
|
09,690 xcecd_worker_utils.c:352 Job ID 1-0-1790907905-32768[2] running - sending status to controller
|
|
```
|
|
`xcecli jobs list` already showed the job RUN, `xcecli jobs kill` answered `job <...> is being <kill>` (apiserver `overallStatus accepted`), but the worker had not yet added the job to its worker list, dropped the request, and the job then ran to completion. Same for a plain job (1-0-1790907905-0: dispatch 06,876, kill 07,313 dropped, running 09,724).
|
|
|
|
**Expected:** a kill accepted for a job the controller shows as RUN takes effect: the worker applies it once the job is registered (or the controller retries / reports the kill as not delivered).
|
|
**Impact:** wrong result: a kill issued in the first seconds after dispatch is reported as accepted but silently lost; the job keeps running. The lookup branch is shared by the real worker (`xcecd`), not only `xcecd_virtual` (real worker not tested).
|
|
**Found by:** test (live verification of bug-1905, 2026-10-01)
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (src/build.jai)
|
|
- Code: scheduler-framework @ 8ac637ad-dirty
|
|
- Deployment: 3-node HA (xce-deploy-ha) + scheduler + apiserver + one xcecd_virtual worker, single host
|
|
|
|
## Reproduction
|
|
1. Deploy with xce-deploy-ha + `start-all.sh`; `xcecli login -u admin -p admin --url http://localhost:8080`.
|
|
2. `xcecli jobs submit -J plain sleep 600 </dev/null`; as soon as `xcecli jobs list` shows RUN (about 1 s), `xcecli jobs kill <id> </dev/null`.
|
|
3. The worker log shows `Failed to find worker for job ID`, and the job stays RUN.
|
|
- Frequency: 2/2 kills sent within ~1.5 s of dispatch, 2026-10-01
|
|
- Evidence: /home/claude/tmp/kill-race/worker.log
|
|
|
|
## Root cause
|
|
The controller forwards a job-control signal as soon as the job has an allocation (`src/controller/job.c` job-control handler -> `xce_jobctrl_req`). The worker answers the start request by queuing it on its thread pool (`controller_connector.c` start handler -> `submit_fork_task`) and only adds the job to its worker list when `xcecd_process_command` runs (`xcecd.c`), which for the first jobs after a worker starts was ~3 s later; the real worker can also only signal a job once the job's IPC client has connected. `handle_jobcontrol` (`controller_connector.c`) dropped a signal for a job not in the list ("Failed to find worker", -1) and also for a listed job still PENDING/STARTING, so the kill was lost while the API had already reported it accepted.
|
|
|
|
## Fix
|
|
New `src/worker/src/pending_signal.c` (`include/pending_signal.h`): a thread-safe table of jobs whose start the worker has accepted. The start handler calls `pending_signal_start()` before queuing the start; `handle_jobcontrol` keeps a signal for a queued or PENDING/STARTING job with `pending_signal_record()` and replies success; `update_worker_status()` takes it on the STARTING -> RUNNING transition and delivers it (virtual: `worker->killed_signal`; real: `ipc_master_send_job_signal`, whose client is connected once the job reports RUNNING). `xcecd_job_start_error()` and the DONE/EXITED transitions drop the entry. A signal for a job the worker knows nothing about still fails as before.
|
|
|
|
## Verification
|
|
- Test: `test_jobcontrol_signal_for_queued_start_is_kept` (RED before the fix: `Expected 0 Was -1`) and `test_jobcontrol_signal_for_unknown_job_fails` in `src/worker/testsuite/test_controller_connector.c`; table semantics in the new `src/worker/testsuite/test_pending_signal.c` (6 tests); GREEN after
|
|
- Suites: worker meson suite 15/15 OK (asan, src/build.jai)
|
|
- Live: fresh xce-deploy-ha + xcecli, kills sent as soon as the jobs listed RUN: the worker logged `has not started yet; signal 15 will be delivered when it runs` for `plain` and `arr[2]` 2.3 s before they started, then `delivering signal 15 kept while it was starting`; both ended EXIT while `arr[1]`/`arr[3]` kept running; a later kill of the running `arr[1]` still worked (/home/claude/tmp/bug-1908/fixed-run.log). The real worker's IPC delivery path is not exercised here (no systemd/dbus in this container).
|
|
|
|
## References
|
|
- Related: none
|