91 lines
4.5 KiB
Markdown
91 lines
4.5 KiB
Markdown
---
|
|
id: "bug-1980"
|
|
status: "fixed"
|
|
status_updated: "2026-10-30"
|
|
status_note: "Gated stdin-script read on len(args)==0; regression test + live t1 verification"
|
|
component: "xcecli"
|
|
severity: "hang"
|
|
timestamp: "2026-10-30T01:52:00Z"
|
|
last_seen: "2026-10-30T01:52:00Z"
|
|
occurrences: 1
|
|
files: ["src/frontend/xcecli/cmd/jobs/submit.go"]
|
|
tags: ["cli", "stdin", "jobs-submit", "external-license-resources"]
|
|
related: []
|
|
fix_commits: ["01f61507"]
|
|
summary: "jobs submit with explicit command blocks forever on a held-open stdin pipe"
|
|
---
|
|
|
|
# bug-1980: jobs submit with explicit command blocks forever on a held-open stdin pipe
|
|
|
|
## Description
|
|
**Observed:** a `xcecli jobs submit -q normal -R "rusage[extlic=1]" -- /tmp/e2e/licjob.sh
|
|
sleep 120` process (pid 705795) sat in `unix_stream_data_wait` on fd 0 for the rest of
|
|
the session (found in the user's process list the next day). No RPC was ever sent; the
|
|
job did not exist on the controller. A minimal repro confirms the hang is in the CLI,
|
|
not the caller:
|
|
|
|
```
|
|
timeout 8 xcecli jobs submit -q normal -R "rusage[extlic=1]" -- /tmp/e2e/licjob.sh sleep 15 < <(sleep 30)
|
|
# rc=124 (timeout kill), no output, no job created
|
|
```
|
|
|
|
**Expected:** `jobs submit` with an explicit command on the command line must submit
|
|
regardless of what stdin is attached to (bsub semantics: stdin is the job script only
|
|
when no command is given).
|
|
|
|
**Impact:** any tooling or human invocation that leaves a pipe on submit's stdin
|
|
(heredoc quoting accidents, script wrappers) hangs the CLI forever with no error and
|
|
no job.
|
|
|
|
**Found by:** the user spotted the hung process in their process list the day after
|
|
the esd-recover verification session (the session's own submit probe, pid 705795).
|
|
|
|
## Environment
|
|
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
|
|
- libc: glibc 2.39 (host ldd); xcecli is static Go (CGO_ENABLED=0, "not a dynamic executable")
|
|
- Toolchain: go 1.23.x
|
|
- Build: debug (src/build.dbg), deployed /opt/xce-denix/bin/xcecli
|
|
- Code: working @ 7520cca7 (E2 external-resources branch line)
|
|
- Deployment: t1 test cluster
|
|
|
|
## Reproduction
|
|
1. `timeout 8 xcecli jobs submit -q normal -- sleep 15 < <(sleep 30)` (or any pipe that
|
|
stays open without writing data or EOF)
|
|
2. rc=124, process blocked in `unix_stream_data_wait` on the stdin socket
|
|
- Frequency: deterministic
|
|
- Evidence: /proc/705795/fd trace at discovery (2026-10-30 10:19 local); repro above
|
|
|
|
## Root cause
|
|
`JobSubmitter.ParseCmdArgs` (src/frontend/xcecli/cmd/jobs/submit.go:105-114) gated on
|
|
"stdin is not a char device" and then `io.ReadAll(os.Stdin)` **before** looking at the
|
|
command args: any non-terminal stdin was treated as a potential job script
|
|
(`parseConfigFile`, the `#xce job submit` script feature). A pipe held open without
|
|
data or EOF blocks `io.ReadAll` forever. The command args present in the same
|
|
invocation are never considered — a silent-open stdin wins over the explicit command,
|
|
and had data arrived it would have been reinterpreted as a config script next to it.
|
|
|
|
## Fix
|
|
Gate the stdin-script read on `len(args) == 0`: stdin is consumed as a job script only
|
|
when no command is given on the command line; an explicit command wins and stdin is
|
|
never read (`submit.go`, ParseCmdArgs). This matches the bsub semantics the CLI's help
|
|
text claims (`command args override config file`), removes the unbounded block, and
|
|
removes the data-ambiguity (stdin bytes silently merged with an explicit command).
|
|
|
|
## Verification
|
|
- Test: `TestParseCmdArgs_ExplicitCommandIgnoresHeldOpenStdin`
|
|
(src/frontend/xcecli/cmd/jobs/submit_test.go) — a held-open pipe as os.Stdin plus
|
|
explicit command args; RED pre-fix (test panics on the 30s timeout guard inside
|
|
`io.ReadAll(os.Stdin)`, confirmed), GREEN post-fix (0.00s).
|
|
- Full suite: `go test ./...` in src/frontend/xcecli — 11/11 packages ok.
|
|
- Live (t1, fixed binary): `jobs submit ... -- licjob.sh sleep 15 < <(sleep 30)` →
|
|
rc=0, Job submitted instantly, RUN→DONE, pool marker released (pre-fix rc=124).
|
|
- Stdin-script path intact: `printf '#xce job submit -q normal\nsleep 5\n' |
|
|
xcecli jobs submit` → submitted, DONE (parseConfigFile's os.Exit(0) path cannot be
|
|
unit-tested in-process; covered live).
|
|
- Suites: `go test ./...` in src/frontend/xcecli — 11/11 packages ok (cmd/jobs,
|
|
cmd/users, cmd/clusters, cmd/hosts, common/format, common/output, common/request,
|
|
filters, models, and the remaining two); apiserver/mcp modules untouched by the fix.
|
|
|
|
## References
|
|
- Found while verifying xceesd kill/restart recover for the external-resources
|
|
manual (§11.5 scenario 4); the hung process was that session's own submit probe.
|