chore(buglog): bug-1765 fixed (0f4d23e3); log bug-1894 (kill right after submit: not found)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
0f4d23e3cd
commit
3e6c48a1fe
GPG key ID:
80533536B0EB651C
3 changed files with 171 additions and 20 deletions
|
|
@ -1,16 +1,17 @@
|
|||
---
|
||||
id: "bug-1765"
|
||||
status: "open"
|
||||
status_updated: "2026-09-24"
|
||||
status_note: "frontend tests/regression/run.sh jobid regex rejects dashed jobid format"
|
||||
component: "unknown"
|
||||
severity: "unknown"
|
||||
status: "fixed"
|
||||
status_updated: "2026-09-30"
|
||||
status_note: ""
|
||||
component: "xcecli"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-09-03T14:51:54Z"
|
||||
last_seen: "2026-09-19T09:00:00Z"
|
||||
occurrences: 1
|
||||
files: []
|
||||
files: ["src/frontend/tests/regression/run.sh"]
|
||||
tags: ["regression-suite", "test-infra", "jobid-format", "false-fail", "pre-existing"]
|
||||
related: []
|
||||
fix_commits: ["0f4d23e3"]
|
||||
legacy: true
|
||||
---
|
||||
|
||||
|
|
@ -19,20 +20,33 @@ legacy: true
|
|||
## Description
|
||||
tests/regression/run.sh's 'jobs submit' test FAILs deterministically (28 passed / 1 failed / 1 skipped instead of 29/0/0), with the cascading 'jobs kill' test SKIPped as a result -- reproduced identically regardless of which build system (Makefile vs Meson) produced the xcecli binary under test, confirming it predates and is unrelated to the eliminate-frontend-makefile migration.
|
||||
|
||||
UPDATE (2026-09-29): also found while reproducing: run from a shell whose stdin is an open pipe (a background job, CI), the suite hangs forever at `jobs submit`, because xcecli's submit reads a job script from stdin whenever stdin is a pipe (src/frontend/xcecli/cmd/jobs/submit.go:96).
|
||||
|
||||
## Environment
|
||||
_Not recorded (migrated from buglog.json)._
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 for the controller (nix dev shell); xcecli/xce_apiserver are Go binaries built by meson (go_build)
|
||||
- Toolchain: go (nix dev shell), gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (src/xce.build.bug1871: controller, scheduler, worker, xceadmin, apiserver, xcecli)
|
||||
- Code: reproduced 2026-09-29 at f5c887dc+
|
||||
- Deployment: local 3-node HA controller cluster + xcecd_virtual worker + scheduler + apiserver on :8080 (deploy-ha.sh + start-all.sh, /home/claude/tmp/xce-1765), config_file auth (admin/admin)
|
||||
|
||||
## Reproduction
|
||||
_Not recorded (migrated from buglog.json)._
|
||||
1. Deploy a cluster with an apiserver: `bash src/tools/deploy-ha.sh <builddir>/controller/xce_controller <top> localhost:9991,localhost:9992,localhost:9993 && bash <top>/scripts/start-all.sh`
|
||||
2. `XCE=<builddir>/frontend/xcecli/xcecli URL=http://localhost:8080 bash src/frontend/tests/regression/run.sh </dev/null`
|
||||
- Frequency: deterministic -- 1/1, 2026-09-29: `FAIL jobs submit — unexpected output: Job <1-0-1790741130-0> is submitted to default queue <normal>.`, `SKIP jobs kill`, `Results: 28 passed 1 failed 1 skipped`; without `</dev/null` from a piped shell it hung at `xcecli jobs submit` for 10+ min
|
||||
- Evidence: /home/claude/tmp/xce-1765 (deploy dir, logs/apiserver.log)
|
||||
|
||||
## Root cause
|
||||
CONFIRMED via source + git history. src/frontend/apiserver/controllers/jobs.go:1518 formats the submit-response message with a plain-digit '%d' placeholder ('Job <%d> is submitted...'), but src/frontend/xcecli/cmd/jobs/submit.go:500 post-processes that message client-side, replacing the '<%d>' substring via format.FormatJobID() (src/frontend/xcecli/common/format/jobid.go:40-57), which renders the packed 64-bit job ID as the dashed '{cluster}-{apiserver}-{timestamp}-{sequence}' form (e.g. '1-0-1788446837-0'). tests/regression/run.sh:143's assertion regex ('Job <[0-9]+> is submitted') was never updated for this dashed format. The CLI-side reformatting was introduced by commit d6008ead ('improve frontend jobid display', 2026-05-07) -- nearly four months before this discovery -- so the test has been silently broken (as a false FAIL, not a false PASS) since then, not a new regression.
|
||||
|
||||
## Fix
|
||||
NOT FIXED -- out of scope for the eliminate-frontend-makefile spec's task 3.5 (pure verification task, no file changes permitted). Fix would be updating run.sh:143's regex to accept the dashed job-ID format (e.g. 'Job <[0-9-]+> is submitted' or delegating to the same format.FormatJobID() pattern). A second, independent, lower-severity issue was found in the same file during this investigation: run.sh:155's assert_not_contains 'jobs list submit_time not dash' calls grep -qF "- " without a '--' separator, so GNU grep treats the leading-dash argument as an invalid option and errors out -- this coincidentally still yields the intended PASS (grep's error-exit is indistinguishable here from 'no match'), so it doesn't affect reported counts, but isn't actually testing what it appears to test.
|
||||
src/frontend/tests/regression/run.sh: the submit check matches the dashed ID (`Job <[0-9]+(-[0-9]+){3}> is submitted`) and extracts the whole ID from between the angle brackets (the old `[0-9]{10,}` would have taken only the timestamp field), which `jobs kill` parses back (format.ParseJobID). The submit command reads `</dev/null` so it never waits on an inherited pipe.
|
||||
|
||||
## Verification
|
||||
_Not recorded (migrated from buglog.json)._
|
||||
- Test: the suite itself -- before: 28 passed / 1 failed / 1 skipped (and a hang with piped stdin); after: 30 passed / 0 failed / 0 skipped, both with stdin from /dev/null and with stdin an open pipe (`sleep 400 | ...`), against the same live cluster
|
||||
- Suites: n/a -- test-script-only change; `bash -n` clean
|
||||
- Live: manual check that the extracted ID is the job's real ID: `jobs list -u all` shows `1-0-1790741567-0 ... kcheck`, and `jobs kill 1-0-1790741567-0` answers `is being <kill>` (an immediate kill right after submit can briefly answer `not found` -- logged separately as bug-1894)
|
||||
|
||||
## References
|
||||
_Not recorded (migrated from buglog.json)._
|
||||
- Fix commit: 0f4d23e3
|
||||
- Related: d6008ead (dashed job-ID display), bug-1894 (submit-to-visible lag)
|
||||
|
|
|
|||
56
.wolf/bugs/bug-1894.md
Normal file
56
.wolf/bugs/bug-1894.md
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
---
|
||||
id: "bug-1894"
|
||||
status: "open"
|
||||
status_updated: "2026-09-30"
|
||||
status_note: "newly logged"
|
||||
component: "apiserver"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-09-30T04:14:02Z"
|
||||
last_seen: "2026-09-30T04:14:02Z"
|
||||
occurrences: 1
|
||||
files: ["src/frontend/apiserver/controllers/jobs.go (job kill lookup)"]
|
||||
tags: ["apiserver", "jobs", "kill", "consistency", "latency"]
|
||||
related: ["bug-1765"]
|
||||
---
|
||||
|
||||
# bug-1894: xce_apiserver: jobs kill right after jobs submit can answer 'job not found' for the accepted job
|
||||
|
||||
## Description
|
||||
**Observed:** `xcecli jobs kill <id>` issued right after `jobs submit` returned that ID can answer that the job does not exist; retried moments later, the same kill succeeds:
|
||||
```
|
||||
Job <1-0-1790741567-0> is submitted to default queue <normal>.
|
||||
job <1-0-1790741567-0> not found
|
||||
... (seconds later)
|
||||
job <1-0-1790741567-0> is being <kill>
|
||||
```
|
||||
**Expected:** once submit has returned a job ID, operations on that ID see the job.
|
||||
**Impact:** minor -- a script that submits and immediately controls a job gets a spurious `not found`; the job itself is fine and appears in `jobs list` under the same ID.
|
||||
**Found by:** manual check while verifying bug-1765's regression-suite fix, 2026-09-29. Suspected (not verified): the apiserver learns of a new job through the controller's asynchronous job notification (OnJobCreated) rather than from the submit reply, so its job store lags the submit acknowledgement by up to a notification round-trip.
|
||||
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (xce.build)
|
||||
- Code: scheduler-framework @ 0f4d23e3-dirty
|
||||
- Deployment: local 3-node HA controller cluster + xcecd_virtual worker + scheduler + apiserver on :8080 (deploy-ha.sh + start-all.sh), config_file auth
|
||||
|
||||
## Reproduction
|
||||
1. Deploy a cluster with an apiserver (deploy-ha.sh + scripts/start-all.sh) and `xcecli login -u admin -p admin --url http://localhost:8080`
|
||||
2. `id=$(xcecli jobs submit -q normal sleep 300 </dev/null | sed -nE 's/.*Job <([^>]+)>.*/\1/p'); xcecli jobs kill "$id"`
|
||||
- Frequency: 2/4 immediate kills answered `not found`, 2026-09-29 (the lag measured 0.01-0.22 s in three timed tries)
|
||||
- Evidence: /home/claude/tmp/xce-1765/logs/apiserver.log (21:12:47 query for job_id 73935322667286528)
|
||||
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
Loading…
Add table
Add a link
Reference in a new issue