docs(buglog): bug-1905 fixed in 8ac637ad; log bug-1908 (kill lost before worker registers job)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
8ac637ad36
commit
7d6a6e6043
3 changed files with 103 additions and 15 deletions
59
.wolf/bugs/bug-1908.md
Normal file
59
.wolf/bugs/bug-1908.md
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
---
|
||||
id: "bug-1908"
|
||||
status: "investigating"
|
||||
status_updated: "2026-10-02"
|
||||
status_note: "worker drops a kill that arrives before the job is in its worker list; controller-side handling of the worker's error (retry or not) not yet traced"
|
||||
component: "worker"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-02T02:27:01Z"
|
||||
last_seen: "2026-10-02T02:27:01Z"
|
||||
occurrences: 1
|
||||
files: ["src/worker/src/controller_connector.c"]
|
||||
tags: ["job-control", "kill", "race", "dispatch"]
|
||||
related: ["bug-1905"]
|
||||
---
|
||||
|
||||
# bug-1908: a kill sent right after dispatch is lost: the worker drops it before registering the job
|
||||
|
||||
## Description
|
||||
**Observed:**
|
||||
```
|
||||
05,850 controller_connector.c:563 Job ID: 1-0-1790907905-32768[2], Command: sleep 600 ... (dispatch received)
|
||||
07,318 controller_connector.c:1546 Received job control request to send SIGINFO for job ID: 1-0-1790907905-32768[2]
|
||||
07,318 controller_connector.c:1586 Failed to find worker for job ID: 1-0-1790907905-32768[2]
|
||||
09,690 xcecd_worker_utils.c:352 Job ID 1-0-1790907905-32768[2] running - sending status to controller
|
||||
```
|
||||
`xcecli jobs list` already showed the job RUN, `xcecli jobs kill` answered `job <...> is being <kill>` (apiserver `overallStatus accepted`), but the worker had not yet added the job to its worker list, dropped the request, and the job then ran to completion. Same for a plain job (1-0-1790907905-0: dispatch 06,876, kill 07,313 dropped, running 09,724).
|
||||
|
||||
**Expected:** a kill accepted for a job the controller shows as RUN takes effect: the worker applies it once the job is registered (or the controller retries / reports the kill as not delivered).
|
||||
**Impact:** wrong result: a kill issued in the first seconds after dispatch is reported as accepted but silently lost; the job keeps running. The lookup branch is shared by the real worker (`xcecd`), not only `xcecd_virtual` (real worker not tested).
|
||||
**Found by:** test (live verification of bug-1905, 2026-10-01)
|
||||
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (src/build.jai)
|
||||
- Code: scheduler-framework @ 8ac637ad-dirty
|
||||
- Deployment: 3-node HA (xce-deploy-ha) + scheduler + apiserver + one xcecd_virtual worker, single host
|
||||
|
||||
## Reproduction
|
||||
1. Deploy with xce-deploy-ha + `start-all.sh`; `xcecli login -u admin -p admin --url http://localhost:8080`.
|
||||
2. `xcecli jobs submit -J plain sleep 600 </dev/null`; as soon as `xcecli jobs list` shows RUN (about 1 s), `xcecli jobs kill <id> </dev/null`.
|
||||
3. The worker log shows `Failed to find worker for job ID`, and the job stays RUN.
|
||||
- Frequency: 2/2 kills sent within ~1.5 s of dispatch, 2026-10-01
|
||||
- Evidence: /home/claude/tmp/kill-race/worker.log
|
||||
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
Loading…
Add table
Add a link
Reference in a new issue