wolf: log bug-1980 (jobs submit stdin block) as fixed
Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen
parent
01f61507ba
commit
a437eff2c9
2 changed files with 117 additions and 0 deletions
|
|
@ -245,6 +245,32 @@
|
|||
"00f81c5a"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1979.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1980",
|
||||
"status": "fixed",
|
||||
"status_updated": "2026-10-30",
|
||||
"status_note": "Gated stdin-script read on len(args)==0; regression test + live t1 verification",
|
||||
"component": "xcecli",
|
||||
"severity": "hang",
|
||||
"timestamp": "2026-10-30T01:52:00Z",
|
||||
"last_seen": "2026-10-30T01:52:00Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "jobs submit with explicit command blocks forever on a held-open stdin pipe\n\n**Observed:** a `xcecli jobs submit -q normal -R \"rusage[extlic=1]\" -- /tmp/e2e/licjob.sh\nsleep 120` process (pid 705795) sat in `unix_stream_data_wait` on fd 0 for the rest of\nthe session (found in the user's process list the next day). No RPC was ever sent; the\njob did not exist on the controller. A minimal repro confirms the hang is in the CLI,\nnot the caller:\n\n```\ntimeout 8 xcecli jobs submit -q normal -R \"rusage[extlic=1]\" -- /tmp/e2e/licjob.sh sleep 15 < <(sleep 30)\n# rc=124 (timeout kill), no output, no job created\n```\n\n**Expected:** `jobs submit` with an explicit command on the command line must submit\nregardless of what stdin is attached to (bsub semantics: stdin is the job script only\nwhen no command is given).\n\n**Impact:** any tooling or human invocation that leaves a pipe on submit's stdin\n(heredoc quoting accidents, script wrappers) hangs the CLI forever with no error and\nno job.\n\n**Found by:** the user spotted the hung process in their process list the day after\nthe esd-recover verification session (the session's own submit probe, pid 705795).",
|
||||
"file": "src/frontend/xcecli/cmd/jobs/submit.go",
|
||||
"root_cause": "`JobSubmitter.ParseCmdArgs` (src/frontend/xcecli/cmd/jobs/submit.go:105-114) gated on\n\"stdin is not a char device\" and then `io.ReadAll(os.Stdin)` **before** looking at the\ncommand args: any non-terminal stdin was treated as a potential job script\n(`parseConfigFile`, the `#xce job submit` script feature). A pipe held open without\ndata or EOF blocks `io.ReadAll` forever. The command args present in the same\ninvocation are never considered — a silent-open stdin wins over the explicit command,\nand had data arrived it would have been reinterpreted as a config script next to it.",
|
||||
"fix": "Gate the stdin-script read on `len(args) == 0`: stdin is consumed as a job script only\nwhen no command is given on the command line; an explicit command wins and stdin is\nnever read (`submit.go`, ParseCmdArgs). This matches the bsub semantics the CLI's help\ntext claims (`command args override config file`), removes the unbounded block, and\nremoves the data-ambiguity (stdin bytes silently merged with an explicit command).\n\nVerification:\n- Test: `TestParseCmdArgs_ExplicitCommandIgnoresHeldOpenStdin`\n (src/frontend/xcecli/cmd/jobs/submit_test.go) — a held-open pipe as os.Stdin plus\n explicit command args; RED pre-fix (test panics on the 30s timeout guard inside\n `io.ReadAll(os.Stdin)`, confirmed), GREEN post-fix (0.00s).\n- Full suite: `go test ./...` in src/frontend/xcecli — 11/11 packages ok.\n- Live (t1, fixed binary): `jobs submit ... -- licjob.sh sleep 15 < <(sleep 30)` →\n rc=0, Job submitted instantly, RUN→DONE, pool marker released (pre-fix rc=124).\n- Stdin-script path intact: `printf '#xce job submit -q normal\\nsleep 5\\n' |\n xcecli jobs submit` → submitted, DONE (parseConfigFile's os.Exit(0) path cannot be\n unit-tested in-process; covered live).\n- Suites: `go test ./...` in src/frontend/xcecli — 11/11 packages ok (cmd/jobs,\n cmd/users, cmd/clusters, cmd/hosts, common/format, common/output, common/request,\n filters, models, and the remaining two); apiserver/mcp modules untouched by the fix.",
|
||||
"tags": [
|
||||
"cli",
|
||||
"stdin",
|
||||
"jobs-submit",
|
||||
"external-license-resources"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"fix_commits": [
|
||||
"01f61507"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1980.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
91
.wolf/bugs/bug-1980.md
Normal file
91
.wolf/bugs/bug-1980.md
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
---
|
||||
id: "bug-1980"
|
||||
status: "fixed"
|
||||
status_updated: "2026-10-30"
|
||||
status_note: "Gated stdin-script read on len(args)==0; regression test + live t1 verification"
|
||||
component: "xcecli"
|
||||
severity: "hang"
|
||||
timestamp: "2026-10-30T01:52:00Z"
|
||||
last_seen: "2026-10-30T01:52:00Z"
|
||||
occurrences: 1
|
||||
files: ["src/frontend/xcecli/cmd/jobs/submit.go"]
|
||||
tags: ["cli", "stdin", "jobs-submit", "external-license-resources"]
|
||||
related: []
|
||||
fix_commits: ["01f61507"]
|
||||
summary: "jobs submit with explicit command blocks forever on a held-open stdin pipe"
|
||||
---
|
||||
|
||||
# bug-1980: jobs submit with explicit command blocks forever on a held-open stdin pipe
|
||||
|
||||
## Description
|
||||
**Observed:** a `xcecli jobs submit -q normal -R "rusage[extlic=1]" -- /tmp/e2e/licjob.sh
|
||||
sleep 120` process (pid 705795) sat in `unix_stream_data_wait` on fd 0 for the rest of
|
||||
the session (found in the user's process list the next day). No RPC was ever sent; the
|
||||
job did not exist on the controller. A minimal repro confirms the hang is in the CLI,
|
||||
not the caller:
|
||||
|
||||
```
|
||||
timeout 8 xcecli jobs submit -q normal -R "rusage[extlic=1]" -- /tmp/e2e/licjob.sh sleep 15 < <(sleep 30)
|
||||
# rc=124 (timeout kill), no output, no job created
|
||||
```
|
||||
|
||||
**Expected:** `jobs submit` with an explicit command on the command line must submit
|
||||
regardless of what stdin is attached to (bsub semantics: stdin is the job script only
|
||||
when no command is given).
|
||||
|
||||
**Impact:** any tooling or human invocation that leaves a pipe on submit's stdin
|
||||
(heredoc quoting accidents, script wrappers) hangs the CLI forever with no error and
|
||||
no job.
|
||||
|
||||
**Found by:** the user spotted the hung process in their process list the day after
|
||||
the esd-recover verification session (the session's own submit probe, pid 705795).
|
||||
|
||||
## Environment
|
||||
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
|
||||
- libc: glibc 2.39 (host ldd); xcecli is static Go (CGO_ENABLED=0, "not a dynamic executable")
|
||||
- Toolchain: go 1.23.x
|
||||
- Build: debug (src/build.dbg), deployed /opt/xce-denix/bin/xcecli
|
||||
- Code: working @ 7520cca7 (E2 external-resources branch line)
|
||||
- Deployment: t1 test cluster
|
||||
|
||||
## Reproduction
|
||||
1. `timeout 8 xcecli jobs submit -q normal -- sleep 15 < <(sleep 30)` (or any pipe that
|
||||
stays open without writing data or EOF)
|
||||
2. rc=124, process blocked in `unix_stream_data_wait` on the stdin socket
|
||||
- Frequency: deterministic
|
||||
- Evidence: /proc/705795/fd trace at discovery (2026-10-30 10:19 local); repro above
|
||||
|
||||
## Root cause
|
||||
`JobSubmitter.ParseCmdArgs` (src/frontend/xcecli/cmd/jobs/submit.go:105-114) gated on
|
||||
"stdin is not a char device" and then `io.ReadAll(os.Stdin)` **before** looking at the
|
||||
command args: any non-terminal stdin was treated as a potential job script
|
||||
(`parseConfigFile`, the `#xce job submit` script feature). A pipe held open without
|
||||
data or EOF blocks `io.ReadAll` forever. The command args present in the same
|
||||
invocation are never considered — a silent-open stdin wins over the explicit command,
|
||||
and had data arrived it would have been reinterpreted as a config script next to it.
|
||||
|
||||
## Fix
|
||||
Gate the stdin-script read on `len(args) == 0`: stdin is consumed as a job script only
|
||||
when no command is given on the command line; an explicit command wins and stdin is
|
||||
never read (`submit.go`, ParseCmdArgs). This matches the bsub semantics the CLI's help
|
||||
text claims (`command args override config file`), removes the unbounded block, and
|
||||
removes the data-ambiguity (stdin bytes silently merged with an explicit command).
|
||||
|
||||
## Verification
|
||||
- Test: `TestParseCmdArgs_ExplicitCommandIgnoresHeldOpenStdin`
|
||||
(src/frontend/xcecli/cmd/jobs/submit_test.go) — a held-open pipe as os.Stdin plus
|
||||
explicit command args; RED pre-fix (test panics on the 30s timeout guard inside
|
||||
`io.ReadAll(os.Stdin)`, confirmed), GREEN post-fix (0.00s).
|
||||
- Full suite: `go test ./...` in src/frontend/xcecli — 11/11 packages ok.
|
||||
- Live (t1, fixed binary): `jobs submit ... -- licjob.sh sleep 15 < <(sleep 30)` →
|
||||
rc=0, Job submitted instantly, RUN→DONE, pool marker released (pre-fix rc=124).
|
||||
- Stdin-script path intact: `printf '#xce job submit -q normal\nsleep 5\n' |
|
||||
xcecli jobs submit` → submitted, DONE (parseConfigFile's os.Exit(0) path cannot be
|
||||
unit-tested in-process; covered live).
|
||||
- Suites: `go test ./...` in src/frontend/xcecli — 11/11 packages ok (cmd/jobs,
|
||||
cmd/users, cmd/clusters, cmd/hosts, common/format, common/output, common/request,
|
||||
filters, models, and the remaining two); apiserver/mcp modules untouched by the fix.
|
||||
|
||||
## References
|
||||
- Found while verifying xceesd kill/restart recover for the external-resources
|
||||
manual (§11.5 scenario 4); the hung process was that session's own submit probe.
|
||||
Loading…
Add table
Add a link
Reference in a new issue