56 lines
2.7 KiB
Markdown
56 lines
2.7 KiB
Markdown
---
|
|
id: "bug-1894"
|
|
status: "open"
|
|
status_updated: "2026-09-30"
|
|
status_note: "newly logged"
|
|
component: "apiserver"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-09-30T04:14:02Z"
|
|
last_seen: "2026-09-30T04:14:02Z"
|
|
occurrences: 1
|
|
files: ["src/frontend/apiserver/controllers/jobs.go (job kill lookup)"]
|
|
tags: ["apiserver", "jobs", "kill", "consistency", "latency"]
|
|
related: ["bug-1765"]
|
|
---
|
|
|
|
# bug-1894: xce_apiserver: jobs kill right after jobs submit can answer 'job not found' for the accepted job
|
|
|
|
## Description
|
|
**Observed:** `xcecli jobs kill <id>` issued right after `jobs submit` returned that ID can answer that the job does not exist; retried moments later, the same kill succeeds:
|
|
```
|
|
Job <1-0-1790741567-0> is submitted to default queue <normal>.
|
|
job <1-0-1790741567-0> not found
|
|
... (seconds later)
|
|
job <1-0-1790741567-0> is being <kill>
|
|
```
|
|
**Expected:** once submit has returned a job ID, operations on that ID see the job.
|
|
**Impact:** minor -- a script that submits and immediately controls a job gets a spurious `not found`; the job itself is fine and appears in `jobs list` under the same ID.
|
|
**Found by:** manual check while verifying bug-1765's regression-suite fix, 2026-09-29. Suspected (not verified): the apiserver learns of a new job through the controller's asynchronous job notification (OnJobCreated) rather than from the submit reply, so its job store lags the submit acknowledgement by up to a notification round-trip.
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (xce.build)
|
|
- Code: scheduler-framework @ 0f4d23e3-dirty
|
|
- Deployment: local 3-node HA controller cluster + xcecd_virtual worker + scheduler + apiserver on :8080 (deploy-ha.sh + start-all.sh), config_file auth
|
|
|
|
## Reproduction
|
|
1. Deploy a cluster with an apiserver (deploy-ha.sh + scripts/start-all.sh) and `xcecli login -u admin -p admin --url http://localhost:8080`
|
|
2. `id=$(xcecli jobs submit -q normal sleep 300 </dev/null | sed -nE 's/.*Job <([^>]+)>.*/\1/p'); xcecli jobs kill "$id"`
|
|
- Frequency: 2/4 immediate kills answered `not found`, 2026-09-29 (the lag measured 0.01-0.22 s in three timed tries)
|
|
- Evidence: /home/claude/tmp/xce-1765/logs/apiserver.log (21:12:47 query for job_id 73935322667286528)
|
|
|
|
## Root cause
|
|
TODO: required for fixed/partial
|
|
|
|
## Fix
|
|
TODO: required for fixed/partial
|
|
|
|
## Verification
|
|
- Test: TODO: name + file; RED before the fix, GREEN after
|
|
- Suites: TODO: suite results and build profile
|
|
- Live: TODO: soak/deploy run, or n/a and why
|
|
|
|
## References
|
|
- Related: none
|