From e1c68e9c7efb1d714b6e3a8736fe845b8c396667 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Wed, 30 Sep 2026 04:24:48 +0000 Subject: [PATCH] chore(buglog): bug-1894 fixed (141ac22c) Co-Authored-By: Claude Opus 5.5 (1M context) --- .wolf/bugs/bug-1894.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/.wolf/bugs/bug-1894.md b/.wolf/bugs/bug-1894.md index 6846bbf7..773319b7 100644 --- a/.wolf/bugs/bug-1894.md +++ b/.wolf/bugs/bug-1894.md @@ -1,16 +1,17 @@ --- id: "bug-1894" -status: "open" +status: "fixed" status_updated: "2026-09-30" -status_note: "newly logged" +status_note: "" component: "apiserver" severity: "wrong-result" timestamp: "2026-09-30T04:14:02Z" last_seen: "2026-09-30T04:14:02Z" occurrences: 1 -files: ["src/frontend/apiserver/controllers/jobs.go (job kill lookup)"] +files: ["src/frontend/apiserver/services/job_service.go (SubmitJob)", "src/frontend/apiserver/backend/accepted.go", "src/frontend/apiserver/backend/handler.go (dispatchJobArray)", "src/frontend/apiserver/store/store.go"] tags: ["apiserver", "jobs", "kill", "consistency", "latency"] related: ["bug-1765"] +fix_commits: ["141ac22c"] --- # bug-1894: xce_apiserver: jobs kill right after jobs submit can answer 'job not found' for the accepted job @@ -27,6 +28,8 @@ job <1-0-1790741567-0> is being **Impact:** minor -- a script that submits and immediately controls a job gets a spurious `not found`; the job itself is fine and appears in `jobs list` under the same ID. **Found by:** manual check while verifying bug-1765's regression-suite fix, 2026-09-29. Suspected (not verified): the apiserver learns of a new job through the controller's asynchronous job notification (OnJobCreated) rather than from the submit reply, so its job store lags the submit acknowledgement by up to a notification round-trip. +UPDATE (2026-09-29): suspicion confirmed -- backend.SubmitJob (clib.go) documented that the store is updated only by the backend's OnJobCreated push, not on submit. + ## Environment - OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 - libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) @@ -42,15 +45,16 @@ job <1-0-1790741567-0> is being - Evidence: /home/claude/tmp/xce-1765/logs/apiserver.log (21:12:47 query for job_id 73935322667286528) ## Root cause -TODO: required for fixed/partial +backend.SubmitJob (src/frontend/apiserver/backend/clib.go) gets the new job's ID from the controller's submit reply but, by design, did not write the job store: store.Jobs was filled only by the controller's asynchronous OnJobCreated / array-push notification. The controller sends both from the same point (after the job is committed and applied), so the submit reply and the notification race. Job control (ControlJobByID) validates the job's status and owner against store.Jobs, so a kill that arrived at the apiserver before the notification found no record and answered `job not found`. ## Fix -TODO: required for fixed/partial +services.SubmitJob (src/frontend/apiserver/services/job_service.go), the single submit path for CreateJob and RestartJobByID, now calls the new backend.RecordAcceptedJob after a successful backend submit. It stores one record per element (base+i for an array, matching the controller's expansion) with status PEND and a submit time, using the new atomic store.DataStore.PutIfAbsent, and marks it models.JobInfo.Provisional (json "-", so never a query column). A notification that already arrived is never overwritten; OnJobCreated/OnJobUpdated (Put) replace the provisional record when they arrive, and dispatchJobArray's duplicate check now skips only non-provisional records, so an array push still replaces provisional elements. Safe because the controller replies only after commit + apply, so the recorded job exists. ## Verification -- Test: TODO: name + file; RED before the fix, GREEN after -- Suites: TODO: suite results and build profile -- Live: TODO: soak/deploy run, or n/a and why +- Test: src/frontend/apiserver/backend/accepted_test.go -- TestRecordAcceptedJob_VisibleImmediately, _DoesNotOverwriteBackendRecord, _BackendPushReplacesProvisional, _ArrayElementsReplacedByPush; plus TestDataStore_PutIfAbsent (store/store_test.go). RED before the fix (did not compile: RecordAcceptedJob / Provisional / PutIfAbsent undefined), GREEN after +- Suites: apiserver Go packages store, utils, controllers, backend, backend/auth and all xcecli packages ok (CGO_ENABLED=0 go test via xce-build.sh apiserver) +- Live: 3-node HA cluster + worker + scheduler + new apiserver (2026-09-29): 20/20 `jobs kill` issued immediately after `jobs submit` answered `is being `, 0 `not found` (before: 2/4 not found); an array job (-J 'arr[1-3]') listed all 3 elements PEND immediately, then RUN with exec host after the array push (`job array push received and expanded` logged once), and killing it reached all 3 elements; frontend regression suite 30/30 ## References -- Related: none +- Fix commit: 141ac22c +- Related: bug-1765 (found while verifying it)