chore(buglog): bug-1894 fixed (141ac22c)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-09-30 04:24:48 +00:00

5.1 KiB

Raw Blame History

id status status_updated status_note component severity timestamp last_seen occurrences files tags related fix_commits

bug-1894

fixed

2026-09-30

apiserver

wrong-result

2026-09-30T04:14:02Z

src/frontend/apiserver/services/job_service.go (SubmitJob)

src/frontend/apiserver/backend/accepted.go

src/frontend/apiserver/backend/handler.go (dispatchJobArray)

src/frontend/apiserver/store/store.go

apiserver

jobs

kill

consistency

latency

bug-1765

141ac22c

bug-1894: xce_apiserver: jobs kill right after jobs submit can answer 'job not found' for the accepted job

Description

Observed: xcecli jobs kill <id> issued right after jobs submit returned that ID can answer that the job does not exist; retried moments later, the same kill succeeds:

Job <1-0-1790741567-0> is submitted to default queue <normal>.
job <1-0-1790741567-0> not found
... (seconds later)
job <1-0-1790741567-0> is being <kill>

Expected: once submit has returned a job ID, operations on that ID see the job. Impact: minor -- a script that submits and immediately controls a job gets a spurious not found; the job itself is fine and appears in jobs list under the same ID. Found by: manual check while verifying bug-1765's regression-suite fix, 2026-09-29. Suspected (not verified): the apiserver learns of a new job through the controller's asynchronous job notification (OnJobCreated) rather than from the submit reply, so its job store lags the submit acknowledgement by up to a notification round-trip.

UPDATE (2026-09-29): suspicion confirmed -- backend.SubmitJob (clib.go) documented that the store is updated only by the backend's OnJobCreated push, not on submit.

Environment

OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
Build: asan (xce.build)
Code: scheduler-framework @ 0f4d23e3-dirty
Deployment: local 3-node HA controller cluster + xcecd_virtual worker + scheduler + apiserver on :8080 (deploy-ha.sh + start-all.sh), config_file auth

Reproduction

Deploy a cluster with an apiserver (deploy-ha.sh + scripts/start-all.sh) and xcecli login -u admin -p admin --url http://localhost:8080
id=$(xcecli jobs submit -q normal sleep 300 </dev/null | sed -nE 's/.*Job <([^>]+)>.*/\1/p'); xcecli jobs kill "$id"

Frequency: 2/4 immediate kills answered not found, 2026-09-29 (the lag measured 0.01-0.22 s in three timed tries)
Evidence: /home/claude/tmp/xce-1765/logs/apiserver.log (21:12:47 query for job_id 73935322667286528)

Root cause

backend.SubmitJob (src/frontend/apiserver/backend/clib.go) gets the new job's ID from the controller's submit reply but, by design, did not write the job store: store.Jobs was filled only by the controller's asynchronous OnJobCreated / array-push notification. The controller sends both from the same point (after the job is committed and applied), so the submit reply and the notification race. Job control (ControlJobByID) validates the job's status and owner against store.Jobs, so a kill that arrived at the apiserver before the notification found no record and answered job not found.

Fix

services.SubmitJob (src/frontend/apiserver/services/job_service.go), the single submit path for CreateJob and RestartJobByID, now calls the new backend.RecordAcceptedJob after a successful backend submit. It stores one record per element (base+i for an array, matching the controller's expansion) with status PEND and a submit time, using the new atomic store.DataStore.PutIfAbsent, and marks it models.JobInfo.Provisional (json "-", so never a query column). A notification that already arrived is never overwritten; OnJobCreated/OnJobUpdated (Put) replace the provisional record when they arrive, and dispatchJobArray's duplicate check now skips only non-provisional records, so an array push still replaces provisional elements. Safe because the controller replies only after commit + apply, so the recorded job exists.

Verification

Test: src/frontend/apiserver/backend/accepted_test.go -- TestRecordAcceptedJob_VisibleImmediately, _DoesNotOverwriteBackendRecord, _BackendPushReplacesProvisional, _ArrayElementsReplacedByPush; plus TestDataStore_PutIfAbsent (store/store_test.go). RED before the fix (did not compile: RecordAcceptedJob / Provisional / PutIfAbsent undefined), GREEN after
Suites: apiserver Go packages store, utils, controllers, backend, backend/auth and all xcecli packages ok (CGO_ENABLED=0 go test via xce-build.sh apiserver)
Live: 3-node HA cluster + worker + scheduler + new apiserver (2026-09-29): 20/20 jobs kill issued immediately after jobs submit answered is being <kill>, 0 not found (before: 2/4 not found); an array job (-J 'arr[1-3]') listed all 3 elements PEND immediately, then RUN with exec host after the array push (job array push received and expanded logged once), and killing it reached all 3 elements; frontend regression suite 30/30

References

Fix commit: 141ac22c
Related: bug-1765 (found while verifying it)

5.1 KiB Raw Blame History