chore(buglog): bug-1765 fixed (0f4d23e3); log bug-1894 (kill right after submit: not found)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-09-30 04:14:19 +00:00

4.4 KiB

Raw Blame History

id status status_updated status_note component severity timestamp last_seen occurrences files tags related fix_commits legacy

bug-1765

fixed

2026-09-30

xcecli

wrong-result

2026-09-03T14:51:54Z

2026-09-19T09:00:00Z

src/frontend/tests/regression/run.sh

regression-suite

test-infra

jobid-format

false-fail

pre-existing

0f4d23e3

true

bug-1765: tests/regression/run.sh's 'jobs submit' test FAILs deterministically (28 passed / 1 failed / 1…

Description

tests/regression/run.sh's 'jobs submit' test FAILs deterministically (28 passed / 1 failed / 1 skipped instead of 29/0/0), with the cascading 'jobs kill' test SKIPped as a result -- reproduced identically regardless of which build system (Makefile vs Meson) produced the xcecli binary under test, confirming it predates and is unrelated to the eliminate-frontend-makefile migration.

UPDATE (2026-09-29): also found while reproducing: run from a shell whose stdin is an open pipe (a background job, CI), the suite hangs forever at jobs submit, because xcecli's submit reads a job script from stdin whenever stdin is a pipe (src/frontend/xcecli/cmd/jobs/submit.go:96).

Environment

OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
libc: glibc 2.42-47 for the controller (nix dev shell); xcecli/xce_apiserver are Go binaries built by meson (go_build)
Toolchain: go (nix dev shell), gcc (GCC) 15.2.0, meson 1.9.1
Build: asan (src/xce.build.bug1871: controller, scheduler, worker, xceadmin, apiserver, xcecli)
Code: reproduced 2026-09-29 at f5c887dc+
Deployment: local 3-node HA controller cluster + xcecd_virtual worker + scheduler + apiserver on :8080 (deploy-ha.sh + start-all.sh, /home/claude/tmp/xce-1765), config_file auth (admin/admin)

Reproduction

Deploy a cluster with an apiserver: bash src/tools/deploy-ha.sh <builddir>/controller/xce_controller <top> localhost:9991,localhost:9992,localhost:9993 && bash <top>/scripts/start-all.sh
XCE=<builddir>/frontend/xcecli/xcecli URL=http://localhost:8080 bash src/frontend/tests/regression/run.sh </dev/null

Frequency: deterministic -- 1/1, 2026-09-29: FAIL jobs submit — unexpected output: Job <1-0-1790741130-0> is submitted to default queue <normal>., SKIP jobs kill, Results: 28 passed 1 failed 1 skipped; without </dev/null from a piped shell it hung at xcecli jobs submit for 10+ min
Evidence: /home/claude/tmp/xce-1765 (deploy dir, logs/apiserver.log)

Root cause

CONFIRMED via source + git history. src/frontend/apiserver/controllers/jobs.go:1518 formats the submit-response message with a plain-digit '%d' placeholder ('Job <%d> is submitted...'), but src/frontend/xcecli/cmd/jobs/submit.go:500 post-processes that message client-side, replacing the '<%d>' substring via format.FormatJobID() (src/frontend/xcecli/common/format/jobid.go:40-57), which renders the packed 64-bit job ID as the dashed '{cluster}-{apiserver}-{timestamp}-{sequence}' form (e.g. '1-0-1788446837-0'). tests/regression/run.sh:143's assertion regex ('Job <[0-9]+> is submitted') was never updated for this dashed format. The CLI-side reformatting was introduced by commit d6008ead ('improve frontend jobid display', 2026-05-07) -- nearly four months before this discovery -- so the test has been silently broken (as a false FAIL, not a false PASS) since then, not a new regression.

Fix

src/frontend/tests/regression/run.sh: the submit check matches the dashed ID (Job <[0-9]+(-[0-9]+){3}> is submitted) and extracts the whole ID from between the angle brackets (the old [0-9]{10,} would have taken only the timestamp field), which jobs kill parses back (format.ParseJobID). The submit command reads </dev/null so it never waits on an inherited pipe.

Verification

Test: the suite itself -- before: 28 passed / 1 failed / 1 skipped (and a hang with piped stdin); after: 30 passed / 0 failed / 0 skipped, both with stdin from /dev/null and with stdin an open pipe (sleep 400 | ...), against the same live cluster
Suites: n/a -- test-script-only change; bash -n clean
Live: manual check that the extracted ID is the job's real ID: jobs list -u all shows 1-0-1790741567-0 ... kcheck, and jobs kill 1-0-1790741567-0 answers is being <kill> (an immediate kill right after submit can briefly answer not found -- logged separately as bug-1894)

References

Fix commit: 0f4d23e3
Related: d6008ead (dashed job-ID display), bug-1894 (submit-to-visible lag)

4.4 KiB Raw Blame History