wolf: log bug-1980 (jobs submit stdin block) as fixed
Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen
parent
01f61507ba
commit
a437eff2c9
2 changed files with 117 additions and 0 deletions
|
|
@ -245,6 +245,32 @@
|
||||||
"00f81c5a"
|
"00f81c5a"
|
||||||
],
|
],
|
||||||
"record": ".wolf/bugs/bug-1979.md"
|
"record": ".wolf/bugs/bug-1979.md"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "bug-1980",
|
||||||
|
"status": "fixed",
|
||||||
|
"status_updated": "2026-10-30",
|
||||||
|
"status_note": "Gated stdin-script read on len(args)==0; regression test + live t1 verification",
|
||||||
|
"component": "xcecli",
|
||||||
|
"severity": "hang",
|
||||||
|
"timestamp": "2026-10-30T01:52:00Z",
|
||||||
|
"last_seen": "2026-10-30T01:52:00Z",
|
||||||
|
"occurrences": 1,
|
||||||
|
"error_message": "jobs submit with explicit command blocks forever on a held-open stdin pipe\n\n**Observed:** a `xcecli jobs submit -q normal -R \"rusage[extlic=1]\" -- /tmp/e2e/licjob.sh\nsleep 120` process (pid 705795) sat in `unix_stream_data_wait` on fd 0 for the rest of\nthe session (found in the user's process list the next day). No RPC was ever sent; the\njob did not exist on the controller. A minimal repro confirms the hang is in the CLI,\nnot the caller:\n\n```\ntimeout 8 xcecli jobs submit -q normal -R \"rusage[extlic=1]\" -- /tmp/e2e/licjob.sh sleep 15 < <(sleep 30)\n# rc=124 (timeout kill), no output, no job created\n```\n\n**Expected:** `jobs submit` with an explicit command on the command line must submit\nregardless of what stdin is attached to (bsub semantics: stdin is the job script only\nwhen no command is given).\n\n**Impact:** any tooling or human invocation that leaves a pipe on submit's stdin\n(heredoc quoting accidents, script wrappers) hangs the CLI forever with no error and\nno job.\n\n**Found by:** the user spotted the hung process in their process list the day after\nthe esd-recover verification session (the session's own submit probe, pid 705795).",
|
||||||
|
"file": "src/frontend/xcecli/cmd/jobs/submit.go",
|
||||||
|
"root_cause": "`JobSubmitter.ParseCmdArgs` (src/frontend/xcecli/cmd/jobs/submit.go:105-114) gated on\n\"stdin is not a char device\" and then `io.ReadAll(os.Stdin)` **before** looking at the\ncommand args: any non-terminal stdin was treated as a potential job script\n(`parseConfigFile`, the `#xce job submit` script feature). A pipe held open without\ndata or EOF blocks `io.ReadAll` forever. The command args present in the same\ninvocation are never considered — a silent-open stdin wins over the explicit command,\nand had data arrived it would have been reinterpreted as a config script next to it.",
|
||||||
|
"fix": "Gate the stdin-script read on `len(args) == 0`: stdin is consumed as a job script only\nwhen no command is given on the command line; an explicit command wins and stdin is\nnever read (`submit.go`, ParseCmdArgs). This matches the bsub semantics the CLI's help\ntext claims (`command args override config file`), removes the unbounded block, and\nremoves the data-ambiguity (stdin bytes silently merged with an explicit command).\n\nVerification:\n- Test: `TestParseCmdArgs_ExplicitCommandIgnoresHeldOpenStdin`\n (src/frontend/xcecli/cmd/jobs/submit_test.go) — a held-open pipe as os.Stdin plus\n explicit command args; RED pre-fix (test panics on the 30s timeout guard inside\n `io.ReadAll(os.Stdin)`, confirmed), GREEN post-fix (0.00s).\n- Full suite: `go test ./...` in src/frontend/xcecli — 11/11 packages ok.\n- Live (t1, fixed binary): `jobs submit ... -- licjob.sh sleep 15 < <(sleep 30)` →\n rc=0, Job submitted instantly, RUN→DONE, pool marker released (pre-fix rc=124).\n- Stdin-script path intact: `printf '#xce job submit -q normal\\nsleep 5\\n' |\n xcecli jobs submit` → submitted, DONE (parseConfigFile's os.Exit(0) path cannot be\n unit-tested in-process; covered live).\n- Suites: `go test ./...` in src/frontend/xcecli — 11/11 packages ok (cmd/jobs,\n cmd/users, cmd/clusters, cmd/hosts, common/format, common/output, common/request,\n filters, models, and the remaining two); apiserver/mcp modules untouched by the fix.",
|
||||||
|
"tags": [
|
||||||
|
"cli",
|
||||||
|
"stdin",
|
||||||
|
"jobs-submit",
|
||||||
|
"external-license-resources"
|
||||||
|
],
|
||||||
|
"related_bugs": [],
|
||||||
|
"fix_commits": [
|
||||||
|
"01f61507"
|
||||||
|
],
|
||||||
|
"record": ".wolf/bugs/bug-1980.md"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
|
||||||
91
.wolf/bugs/bug-1980.md
Normal file
91
.wolf/bugs/bug-1980.md
Normal file
|
|
@ -0,0 +1,91 @@
|
||||||
|
---
|
||||||
|
id: "bug-1980"
|
||||||
|
status: "fixed"
|
||||||
|
status_updated: "2026-10-30"
|
||||||
|
status_note: "Gated stdin-script read on len(args)==0; regression test + live t1 verification"
|
||||||
|
component: "xcecli"
|
||||||
|
severity: "hang"
|
||||||
|
timestamp: "2026-10-30T01:52:00Z"
|
||||||
|
last_seen: "2026-10-30T01:52:00Z"
|
||||||
|
occurrences: 1
|
||||||
|
files: ["src/frontend/xcecli/cmd/jobs/submit.go"]
|
||||||
|
tags: ["cli", "stdin", "jobs-submit", "external-license-resources"]
|
||||||
|
related: []
|
||||||
|
fix_commits: ["01f61507"]
|
||||||
|
summary: "jobs submit with explicit command blocks forever on a held-open stdin pipe"
|
||||||
|
---
|
||||||
|
|
||||||
|
# bug-1980: jobs submit with explicit command blocks forever on a held-open stdin pipe
|
||||||
|
|
||||||
|
## Description
|
||||||
|
**Observed:** a `xcecli jobs submit -q normal -R "rusage[extlic=1]" -- /tmp/e2e/licjob.sh
|
||||||
|
sleep 120` process (pid 705795) sat in `unix_stream_data_wait` on fd 0 for the rest of
|
||||||
|
the session (found in the user's process list the next day). No RPC was ever sent; the
|
||||||
|
job did not exist on the controller. A minimal repro confirms the hang is in the CLI,
|
||||||
|
not the caller:
|
||||||
|
|
||||||
|
```
|
||||||
|
timeout 8 xcecli jobs submit -q normal -R "rusage[extlic=1]" -- /tmp/e2e/licjob.sh sleep 15 < <(sleep 30)
|
||||||
|
# rc=124 (timeout kill), no output, no job created
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected:** `jobs submit` with an explicit command on the command line must submit
|
||||||
|
regardless of what stdin is attached to (bsub semantics: stdin is the job script only
|
||||||
|
when no command is given).
|
||||||
|
|
||||||
|
**Impact:** any tooling or human invocation that leaves a pipe on submit's stdin
|
||||||
|
(heredoc quoting accidents, script wrappers) hangs the CLI forever with no error and
|
||||||
|
no job.
|
||||||
|
|
||||||
|
**Found by:** the user spotted the hung process in their process list the day after
|
||||||
|
the esd-recover verification session (the session's own submit probe, pid 705795).
|
||||||
|
|
||||||
|
## Environment
|
||||||
|
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
|
||||||
|
- libc: glibc 2.39 (host ldd); xcecli is static Go (CGO_ENABLED=0, "not a dynamic executable")
|
||||||
|
- Toolchain: go 1.23.x
|
||||||
|
- Build: debug (src/build.dbg), deployed /opt/xce-denix/bin/xcecli
|
||||||
|
- Code: working @ 7520cca7 (E2 external-resources branch line)
|
||||||
|
- Deployment: t1 test cluster
|
||||||
|
|
||||||
|
## Reproduction
|
||||||
|
1. `timeout 8 xcecli jobs submit -q normal -- sleep 15 < <(sleep 30)` (or any pipe that
|
||||||
|
stays open without writing data or EOF)
|
||||||
|
2. rc=124, process blocked in `unix_stream_data_wait` on the stdin socket
|
||||||
|
- Frequency: deterministic
|
||||||
|
- Evidence: /proc/705795/fd trace at discovery (2026-10-30 10:19 local); repro above
|
||||||
|
|
||||||
|
## Root cause
|
||||||
|
`JobSubmitter.ParseCmdArgs` (src/frontend/xcecli/cmd/jobs/submit.go:105-114) gated on
|
||||||
|
"stdin is not a char device" and then `io.ReadAll(os.Stdin)` **before** looking at the
|
||||||
|
command args: any non-terminal stdin was treated as a potential job script
|
||||||
|
(`parseConfigFile`, the `#xce job submit` script feature). A pipe held open without
|
||||||
|
data or EOF blocks `io.ReadAll` forever. The command args present in the same
|
||||||
|
invocation are never considered — a silent-open stdin wins over the explicit command,
|
||||||
|
and had data arrived it would have been reinterpreted as a config script next to it.
|
||||||
|
|
||||||
|
## Fix
|
||||||
|
Gate the stdin-script read on `len(args) == 0`: stdin is consumed as a job script only
|
||||||
|
when no command is given on the command line; an explicit command wins and stdin is
|
||||||
|
never read (`submit.go`, ParseCmdArgs). This matches the bsub semantics the CLI's help
|
||||||
|
text claims (`command args override config file`), removes the unbounded block, and
|
||||||
|
removes the data-ambiguity (stdin bytes silently merged with an explicit command).
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
- Test: `TestParseCmdArgs_ExplicitCommandIgnoresHeldOpenStdin`
|
||||||
|
(src/frontend/xcecli/cmd/jobs/submit_test.go) — a held-open pipe as os.Stdin plus
|
||||||
|
explicit command args; RED pre-fix (test panics on the 30s timeout guard inside
|
||||||
|
`io.ReadAll(os.Stdin)`, confirmed), GREEN post-fix (0.00s).
|
||||||
|
- Full suite: `go test ./...` in src/frontend/xcecli — 11/11 packages ok.
|
||||||
|
- Live (t1, fixed binary): `jobs submit ... -- licjob.sh sleep 15 < <(sleep 30)` →
|
||||||
|
rc=0, Job submitted instantly, RUN→DONE, pool marker released (pre-fix rc=124).
|
||||||
|
- Stdin-script path intact: `printf '#xce job submit -q normal\nsleep 5\n' |
|
||||||
|
xcecli jobs submit` → submitted, DONE (parseConfigFile's os.Exit(0) path cannot be
|
||||||
|
unit-tested in-process; covered live).
|
||||||
|
- Suites: `go test ./...` in src/frontend/xcecli — 11/11 packages ok (cmd/jobs,
|
||||||
|
cmd/users, cmd/clusters, cmd/hosts, common/format, common/output, common/request,
|
||||||
|
filters, models, and the remaining two); apiserver/mcp modules untouched by the fix.
|
||||||
|
|
||||||
|
## References
|
||||||
|
- Found while verifying xceesd kill/restart recover for the external-resources
|
||||||
|
manual (§11.5 scenario 4); the hung process was that session's own submit probe.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue