From f585469cb6b3012d9ec8a27be5936d9e9c321f7d Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Wed, 30 Sep 2026 04:24:19 +0000 Subject: [PATCH] chore(buglog): log bug-1895 (apiserver Go test failures do not fail xce-build.sh) Co-Authored-By: Claude Opus 5.5 (1M context) --- .wolf/buglog.json | 43 +++++++++++++++++++++++++++------ .wolf/bugs/bug-1895.md | 55 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+), 7 deletions(-) create mode 100644 .wolf/bugs/bug-1895.md diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 5d7e169d..1b3c686d 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -16576,18 +16576,18 @@ }, { "id": "bug-1894", - "status": "open", + "status": "fixed", "status_updated": "2026-09-30", - "status_note": "newly logged", + "status_note": "", "component": "apiserver", "severity": "wrong-result", "timestamp": "2026-09-30T04:14:02Z", "last_seen": "2026-09-30T04:14:02Z", "occurrences": 1, - "error_message": "xce_apiserver: jobs kill right after jobs submit can answer 'job not found' for the accepted job\n\n**Observed:** `xcecli jobs kill ` issued right after `jobs submit` returned that ID can answer that the job does not exist; retried moments later, the same kill succeeds:\n```\nJob <1-0-1790741567-0> is submitted to default queue .\njob <1-0-1790741567-0> not found\n... (seconds later)\njob <1-0-1790741567-0> is being \n```\n**Expected:** once submit has returned a job ID, operations on that ID see the job.\n**Impact:** minor -- a script that submits and immediately controls a job gets a spurious `not found`; the job itself is fine and appears in `jobs list` under the same ID.\n**Found by:** manual check while verifying bug-1765's regression-suite fix, 2026-09-29. Suspected (not verified): the apiserver learns of a new job through the controller's asynchronous job notification (OnJobCreated) rather than from the submit reply, so its job store lags the submit acknowledgement by up to a notification round-trip.", - "file": "src/frontend/apiserver/controllers/jobs.go (job kill lookup)", - "root_cause": "TODO: required for fixed/partial", - "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "error_message": "xce_apiserver: jobs kill right after jobs submit can answer 'job not found' for the accepted job\n\n**Observed:** `xcecli jobs kill ` issued right after `jobs submit` returned that ID can answer that the job does not exist; retried moments later, the same kill succeeds:\n```\nJob <1-0-1790741567-0> is submitted to default queue .\njob <1-0-1790741567-0> not found\n... (seconds later)\njob <1-0-1790741567-0> is being \n```\n**Expected:** once submit has returned a job ID, operations on that ID see the job.\n**Impact:** minor -- a script that submits and immediately controls a job gets a spurious `not found`; the job itself is fine and appears in `jobs list` under the same ID.\n**Found by:** manual check while verifying bug-1765's regression-suite fix, 2026-09-29. Suspected (not verified): the apiserver learns of a new job through the controller's asynchronous job notification (OnJobCreated) rather than from the submit reply, so its job store lags the submit acknowledgement by up to a notification round-trip.\n\nUPDATE (2026-09-29): suspicion confirmed -- backend.SubmitJob (clib.go) documented that the store is updated only by the backend's OnJobCreated push, not on submit.", + "file": "src/frontend/apiserver/services/job_service.go (SubmitJob), src/frontend/apiserver/backend/accepted.go, src/frontend/apiserver/backend/handler.go (dispatchJobArray), src/frontend/apiserver/store/store.go", + "root_cause": "backend.SubmitJob (src/frontend/apiserver/backend/clib.go) gets the new job's ID from the controller's submit reply but, by design, did not write the job store: store.Jobs was filled only by the controller's asynchronous OnJobCreated / array-push notification. The controller sends both from the same point (after the job is committed and applied), so the submit reply and the notification race. Job control (ControlJobByID) validates the job's status and owner against store.Jobs, so a kill that arrived at the apiserver before the notification found no record and answered `job not found`.", + "fix": "services.SubmitJob (src/frontend/apiserver/services/job_service.go), the single submit path for CreateJob and RestartJobByID, now calls the new backend.RecordAcceptedJob after a successful backend submit. It stores one record per element (base+i for an array, matching the controller's expansion) with status PEND and a submit time, using the new atomic store.DataStore.PutIfAbsent, and marks it models.JobInfo.Provisional (json \"-\", so never a query column). A notification that already arrived is never overwritten; OnJobCreated/OnJobUpdated (Put) replace the provisional record when they arrive, and dispatchJobArray's duplicate check now skips only non-provisional records, so an array push still replaces provisional elements. Safe because the controller replies only after commit + apply, so the recorded job exists.\n\nVerification:\n- Test: src/frontend/apiserver/backend/accepted_test.go -- TestRecordAcceptedJob_VisibleImmediately, _DoesNotOverwriteBackendRecord, _BackendPushReplacesProvisional, _ArrayElementsReplacedByPush; plus TestDataStore_PutIfAbsent (store/store_test.go). RED before the fix (did not compile: RecordAcceptedJob / Provisional / PutIfAbsent undefined), GREEN after\n- Suites: apiserver Go packages store, utils, controllers, backend, backend/auth and all xcecli packages ok (CGO_ENABLED=0 go test via xce-build.sh apiserver)\n- Live: 3-node HA cluster + worker + scheduler + new apiserver (2026-09-29): 20/20 `jobs kill` issued immediately after `jobs submit` answered `is being `, 0 `not found` (before: 2/4 not found); an array job (-J 'arr[1-3]') listed all 3 elements PEND immediately, then RUN with exec host after the array push (`job array push received and expanded` logged once), and killing it reached all 3 elements; frontend regression suite 30/30", "tags": [ "apiserver", "jobs", @@ -16598,8 +16598,37 @@ "related_bugs": [ "bug-1765" ], - "fix_commits": [], + "fix_commits": [ + "141ac22c" + ], "record": ".wolf/bugs/bug-1894.md" + }, + { + "id": "bug-1895", + "status": "open", + "status_updated": "2026-09-30", + "status_note": "newly logged", + "component": "build", + "severity": "wrong-result", + "timestamp": "2026-09-30T04:23:48Z", + "last_seen": "2026-09-30T04:23:48Z", + "occurrences": 1, + "error_message": "xce-build.sh apiserver/all: a failing apiserver Go test does not fail the build\n\n**Observed:** `bash src/tools/xce-build.sh apiserver asan --incremental` exited 0 although its Go unit tests failed:\n```\nFAIL\tapiserver/backend [build failed]\nFAIL\tapiserver/store [build failed]\n... exit=0\n```\n**Expected:** a failing unit test fails the build (non-zero exit), as for every C suite.\n**Impact:** apiserver Go test failures are silently ignored by /xce-build (and by `all`); a regression in apiserver/store, utils, controllers or backend can pass the build unnoticed. xcecli's tests are unaffected when they are the ones failing, but are skipped entirely when the apiserver tests fail.\n**Found by:** review of the build output while doing bug-1894's RED step, 2026-09-29.", + "file": "src/tools/xce-build.sh (apiserver and all test steps)", + "root_cause": "TODO: required for fixed/partial", + "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "tags": [ + "build", + "xce-build", + "go", + "tests", + "set-e" + ], + "related_bugs": [ + "bug-1894" + ], + "fix_commits": [], + "record": ".wolf/bugs/bug-1895.md" } ] } diff --git a/.wolf/bugs/bug-1895.md b/.wolf/bugs/bug-1895.md new file mode 100644 index 00000000..f40a2ff4 --- /dev/null +++ b/.wolf/bugs/bug-1895.md @@ -0,0 +1,55 @@ +--- +id: "bug-1895" +status: "open" +status_updated: "2026-09-30" +status_note: "newly logged" +component: "build" +severity: "wrong-result" +timestamp: "2026-09-30T04:23:48Z" +last_seen: "2026-09-30T04:23:48Z" +occurrences: 1 +files: ["src/tools/xce-build.sh (apiserver and all test steps)"] +tags: ["build", "xce-build", "go", "tests", "set-e"] +related: ["bug-1894"] +--- + +# bug-1895: xce-build.sh apiserver/all: a failing apiserver Go test does not fail the build + +## Description +**Observed:** `bash src/tools/xce-build.sh apiserver asan --incremental` exited 0 although its Go unit tests failed: +``` +FAIL apiserver/backend [build failed] +FAIL apiserver/store [build failed] +... exit=0 +``` +**Expected:** a failing unit test fails the build (non-zero exit), as for every C suite. +**Impact:** apiserver Go test failures are silently ignored by /xce-build (and by `all`); a regression in apiserver/store, utils, controllers or backend can pass the build unnoticed. xcecli's tests are unaffected when they are the ones failing, but are skipped entirely when the apiserver tests fail. +**Found by:** review of the build output while doing bug-1894's RED step, 2026-09-29. + +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (the nix dev shell's bash running xce-build.sh; the Go tests themselves run CGO_ENABLED=0) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan profile (apiserver component, --incremental), src/xce.build.bug1871 +- Code: scheduler-framework @ 141ac22c-dirty +- Deployment: n/a -- local build script + +## Reproduction +1. Make any apiserver Go test fail (e.g. reference an undefined symbol in a _test.go under src/frontend/apiserver/backend). +2. `bash src/tools/xce-build.sh apiserver asan --incremental; echo $?` -> prints the FAIL lines, then exits 0. +- Frequency: deterministic -- 1/1, 2026-09-29 (bug-1894's RED run) +- Evidence: the mechanism is in src/tools/xce-build.sh lines ~390 and ~437: `cd frontend/apiserver && go test ... && cd ../xcecli && go test ...` under `set -euo pipefail`; bash's errexit ignores a failure of any command in an && list except the last, so a failing apiserver `go test` just short-circuits the list and the script carries on to exit 0 + +## Root cause +TODO: required for fixed/partial + +## Fix +TODO: required for fixed/partial + +## Verification +- Test: TODO: name + file; RED before the fix, GREEN after +- Suites: TODO: suite results and build profile +- Live: TODO: soak/deploy run, or n/a and why + +## References +- Related: none