72 lines
7.5 KiB
Markdown
72 lines
7.5 KiB
Markdown
---
|
|
id: "bug-1936"
|
|
status: "fixed"
|
|
status_updated: "2026-10-10"
|
|
status_note: ""
|
|
component: "apiserver"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-10-10T08:14:36Z"
|
|
last_seen: "2026-10-10T08:14:36Z"
|
|
occurrences: 1
|
|
files: ["src/frontend/apiserver/backend/convert.go", "src/frontend/apiserver/controllers/jobs.go"]
|
|
tags: ["apiserver", "submit", "exdata", "limits"]
|
|
related: []
|
|
fix_commits: ["d01ced9b", "725b1d23", "edad8c73"]
|
|
gitea_issue: 367
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "150c307e8371"
|
|
---
|
|
|
|
# bug-1936: apiserver drops cwd, env, run/resource limits, file transfers and I/O paths of REST job submissions
|
|
|
|
## Description
|
|
**Observed:** (source reading, `.kiro/reviews/project-gap-review-2026-10-10.md` (P0); to be confirmed by a failing test before the fix)
|
|
`modelJobToSubmit()` (`src/frontend/apiserver/backend/convert.go:323-375`) builds the controller's `job_startinfo_t` with only cmdline, execuser, launch_type and pre/post-exec; it never sets `exdata`, `exechome` or `execcwd`:
|
|
```go
|
|
si := (*C.job_startinfo_t)(C.calloc(1, C.sizeof_job_startinfo_t))
|
|
si.cmdline = C.CString(job.Command)
|
|
... // no si.exdata / si.exechome / si.execcwd
|
|
```
|
|
So `--cwd`, `--env`, `--file` transfers, interactive mode, every `*_limit` flag (incl. `-W` run limit), `-i/-o/-e`, asked hosts and begin/end time are kept only in the apiserver's memory and never reach the controller or the worker.
|
|
|
|
**Expected:** Every submit option the apiserver accepts reaches the worker in the job's start info (exdata, exec home, cwd), or is rejected as unsupported.
|
|
**Impact:** Silent wrong result: jobs run without the limits, environment, working directory and I/O paths the user asked for (e.g. a `-W` run limit is never enforced).
|
|
**Found by:** review: `.kiro/reviews/project-gap-review-2026-10-10.md` (P0)
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (xce.build)
|
|
- Code: scheduler-framework @ 94472ee3-dirty
|
|
- Deployment: any (found by source reading; the failing test that confirms it is named in Verification)
|
|
|
|
## Reproduction
|
|
1. Submit through REST/xcecli with `xcecli jobs submit -W 1 --cwd /tmp --env A=1 sleep 120` on any deployment.
|
|
2. Inspect the job's start info at the worker (worker log of the launched unit) or the controller's `job.start_info`: no exdata, no cwd; the job runs past one minute.
|
|
- Frequency: deterministic (by code reading; confirmed by the RED test of the fix)
|
|
- Evidence: the code sites quoted in Description; no logs kept.
|
|
## Root cause
|
|
`modelJobToSubmit()` (`backend/convert.go`) filled only cmdline, execuser, launch_type and pre/post-exec of `job_startinfo_t`; the extended data `processJobRequest()` (`controllers/jobs.go`) builds (env, limits, transfers, interactive) and the cwd stayed in apiserver memory. Pre-existing, origin not bisected.
|
|
|
|
UPDATE (2026-10-10): the remaining gaps: xcecd's job-assignment message to the per-job worker process (`job_assignment_t`, `src/worker/include/uv_ipc.h`) had no cwd or home field, so the job was spawned with no working directory and exechome was never applied; `parse_limits()` (`xcecd_exdata.c`) ignored `nofile_limit`; and the apiserver accepted in/out/err files, out_dir, asked hosts, begin/term time and transfer files, which have no field in the controller's submit data or start info (the worker parses transfer files but never stages them).
|
|
|
|
## Fix
|
|
`backend/convert_pure.go`: new pure `submitStartInfo()` sets exdata from `ExtendData`, execcwd (`ExecCWD`, else `CWD`) and exechome; `rawJobToModel` restores `ExtendData`/`CWD`; new `ValidateExdata()` rejects non-object or >16383-byte exdata (the worker would fail the job). `backend/convert.go`: copies/frees the three fields and reads exdata back for jobs and arrays. `controllers/jobs.go`: validates exdata and no longer writes `"transfer_files": null`. The controller passes start info through unchanged (WAL submit facts encode the whole JobStartInfo).
|
|
|
|
UPDATE (2026-10-10, `725b1d23`, `edad8c73`), per remaining item:
|
|
- `--cwd`: honoured. `job_assignment_t` gains `cwd`/`home`, filled by new `ipc_fill_job_assignment()` (`uv_ipc.c`, always NUL-terminated); new `uv_proc_enter_job_dirs()` (`uv_proc.c`), called from `worker_job_callback()` (`xcecd_worker.c`), enters the cwd before pre-exec, so pre-exec and the job run there. An unusable cwd falls back to `/tmp` with a WARN (xcecli always sends the client's cwd, so failing would break submissions from directories absent on the worker host). The apiserver rejects a relative or >1023-byte cwd.
|
|
- exec home: honoured. New REST field `exec_home` (`models/jobs.go`, absolute, <=1023 bytes) → start info exechome → the worker sets `HOME` (an explicit `--env HOME=` wins).
|
|
- `nofile_limit`: honoured. `parse_limits()` reads it into `res_limits_t.nofile_limit`; `xcecd_cgroups.c` sets `LimitNOFILE` on the job unit (code-read only: needs D-Bus). xcecli has no flag for it.
|
|
- `in_file`, `out_file`, `err_file`, `out_dir`, `asked_hosts`, `begin_time`, `term_time`, `transfer_files`: rejected. New `preCheckSubmitOptions()` (`controllers/jobs.go`) in `CreateJob` returns 400 "job submission option(s) ... not supported" before upload handling or job-id allocation. `-o/-e` were not honoured because the worker's output fetch is locked to its own output path (bug-1508). Swagger regenerated.
|
|
The same options are still copied and dropped on the job restart path: logged as a separate bug.
|
|
|
|
## Verification
|
|
- Test: `TestSubmitStartInfo_CarriesExdataCwdAndHome`, `TestSubmitStartInfo_PrefersExecCWD`, `TestRawJobArrayToModels_RestoresExdataAndCwd`, `TestValidateExdata` (`backend/convert_test.go`), `TestJobHandler_SubmitRejectsNonObjectExtendData` (`controllers/jobs_handler_test.go`): RED before (e.g. `convert_test.go:1272: exdata: got "", want "{\"limits\":{\"run_limit\":60},...}"`), GREEN after.
|
|
- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch, 2026-10-10: all C suites Ok 155 / Fail 0, frontend Go unit tests ok, xceadmin tests ok; apiserver `CGO_ENABLED=0 go test ./...` (incl. middleware, router, cmd) ok
|
|
- Second fix: `test_assignment_carries_cwd_and_home`, `..._enters_cwd_and_sets_home`, `..._unusable_cwd_falls_back`, `..._spawned_after_enter_runs_in_cwd` (new `src/worker/testsuite/test_job_dirs.c`; RED e.g. `test_job_dirs.c:111:test_assignment_carries_cwd_and_home:FAIL: Expected '/data/run' Was ''`, `test_job_dirs.c:207:...: the job did not write pwd.out in cwd`), `test_apiserver_nofile_limit_parsed` (`test_exdata_parse.c`; RED `:148: FAIL: Expected 2048 Was 0`), `TestJobHandler_SubmitRejectsUnsupportedOptions` (RED `jobs_handler_test.go:1631: in_file: status = 200, want 400`), `TestJobHandler_SubmitRejectsBadCwdAndExecHome` (RED `:1655: exec_home=home/u: status = 500, want 400`), `TestJobHandler_SubmitCarriesCwdAndExecHome`, `TestJobHandler_SubmitNofileLimitInExdata` (`controllers/jobs_handler_test.go`); GREEN after. `TestJobHandler_SubmitArrayFileNameSubstitution` now checks %J/%I substitution on `cwd` (out/err files are refused).
|
|
- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch (scheduler-framework @ edad8c73), 2026-10-10: all C suites Ok 158 / Fail 0, frontend Go unit tests (apiserver + xcecli) ok; apiserver `CGO_ENABLED=0 go test ./...` (all packages) ok
|
|
- Live: n/a: unit/fixture-level verification only; no live deployment was run (follow-up)
|
|
|
|
## References
|
|
- Fix commits: `d01ced9b` (first fix), `725b1d23`, `edad8c73` (worktree originals `3e571f25`, `200127bb`)
|
|
- Review: `.kiro/reviews/project-gap-review-2026-10-10.md`
|