diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 9027c1f7..67fe9224 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -18711,18 +18711,18 @@ }, { "id": "bug-1968", - "status": "open", + "status": "fixed", "status_updated": "2026-10-10", - "status_note": "newly logged", + "status_note": "", "component": "apiserver", "severity": "wrong-result", "timestamp": "2026-10-10T11:55:48Z", "last_seen": "2026-10-10T11:55:48Z", "occurrences": 1, - "error_message": "apiserver job restart copies asked hosts, begin/term time and I/O paths it can never pass on\n\n**Observed:** (source reading, 2026-10-10; found while fixing bug-1936)\n`RestartJobByID()` (`src/frontend/apiserver/controllers/jobs.go:832`) builds the new job from the request plus the old job's row, and copies options that bug-1936 now refuses at submit because no field carries them to the controller or worker:\n```go\nif newJob.AskedHosts == \"\" {\n if v, ok := oldJob[\"asked_hosts\"].(string); ok { newJob.AskedHosts = v }\n}\nif v, ok := oldJob[\"begin_time\"].(time.Time); ok { ... } // also term_time,\nif v, ok := oldJob[\"out_dir\"].(string); ok { ... } // in_file, err_file, out_file\n```\n`preCheckJobRestartRequest()` (`:1350`) validates `-m` asked hosts but does not refuse them the way `preCheckSubmitOptions()` (`:1307`, bug-1936) does for submit, so `xcecli jobs restart -m host ...` returns success and the host request is silently dropped.\n\n**Expected:** restart, like submit, either passes each option on to the job or rejects it with 400 (`preCheckSubmitOptions`-style); options copied from the old job that cannot be honoured are not silently kept in apiserver memory only.\n**Impact:** wrong result: a restarted job ignores the asked hosts, begin/term time and I/O paths the user gave, with no error.\n**Found by:** review during the bug-1936 fix, 2026-10-10.", + "error_message": "apiserver job restart copies asked hosts, begin/term time and I/O paths it can never pass on\n\n**Observed:** (source reading, 2026-10-10; found while fixing bug-1936)\n`RestartJobByID()` (`src/frontend/apiserver/controllers/jobs.go:832`) builds the new job from the request plus the old job's row, and copies options that bug-1936 now refuses at submit because no field carries them to the controller or worker:\n```go\nif newJob.AskedHosts == \"\" {\n if v, ok := oldJob[\"asked_hosts\"].(string); ok { newJob.AskedHosts = v }\n}\nif v, ok := oldJob[\"begin_time\"].(time.Time); ok { ... } // also term_time,\nif v, ok := oldJob[\"out_dir\"].(string); ok { ... } // in_file, err_file, out_file\n```\n`preCheckJobRestartRequest()` (`:1350`) validates `-m` asked hosts but does not refuse them the way `preCheckSubmitOptions()` (`:1307`, bug-1936) does for submit, so `xcecli jobs restart -m host ...` returns success and the host request is silently dropped.\n\n**Expected:** restart, like submit, either passes each option on to the job or rejects it with 400 (`preCheckSubmitOptions`-style); options copied from the old job that cannot be honoured are not silently kept in apiserver memory only.\n**Impact:** wrong result: a restarted job ignores the asked hosts, begin/term time and I/O paths the user gave, with no error.\n**Found by:** review during the bug-1936 fix, 2026-10-10.\n\nUPDATE (2026-10-10, during the fix): the same rebuild also lost options the job DOES get. Rows come from `store.StructToOrderMap`, which keeps each field's Go type, so `oldJob[\"chkpnt_info\"].(string)` and `oldJob[\"extend_data\"].(string)` never matched `models.Jsonb`: env/limits (exdata) were dropped on every restart and a checkpointed job never restarted from its checkpoint (`restart_command`, `XCE_JOB_CHECKPOINT_PID`). It also read `oldJob[\"res_req\"]` (the column is `resreq`), dropping the resource requirement, and never copied `exec_home`.", "file": "src/frontend/apiserver/controllers/jobs.go", - "root_cause": "TODO: required for fixed/partial", - "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "root_cause": "`RestartJobByID()` (`src/frontend/apiserver/controllers/jobs.go`) rebuilds the new job by copying fields out of the old job's stored row. Its copy list predates bug-1936: it accepted `asked_hosts` in the request and copied asked hosts, begin/term time, in/out/err files and `out_dir`, none of which any job gets. The list was also never checked against the row's real keys and types: JSONB columns are `models.Jsonb` (not `string`) and the resource requirement key is `resreq` (not `res_req`), so those copies silently did nothing. Pre-existing, origin not bisected.", + "fix": "`controllers/jobs.go`:\n- `preCheckJobRestartRequest()` refuses a non-blank `asked_hosts` with 400 (\"job restart option asked_hosts not supported\"), like `preCheckSubmitOptions()` (bug-1936); the request field stays so the client gets that message, not \"unknown field\". `processJobRestartRequest()` (which only copied asked hosts) is removed.\n- The restart no longer carries over asked hosts, begin/term time, in/out/err files or `out_dir`.\n- New `rowJSON()` reads a JSONB row column as `models.Jsonb` or string; used for `chkpnt_info` and `extend_data`, so a checkpointed job restarts with its restart command and checkpoint pid, and every restart keeps the old job's extended data (env, limits).\n- Copies `resreq` (was `res_req`) and `exec_home`.\n\nVerification:\n- Test: `TestJobHandler_RestartRejectsAskedHosts`, `TestJobHandler_RestartCarriesOnlyHonouredOptions`, `TestJobHandler_RestartFromCheckpoint` (`src/frontend/apiserver/controllers/jobs_handler_test.go`): RED before (`jobs_handler_test.go:1537: status = 200, want 400`; `:1592: restarted job kept asked_hosts, which no job gets` (also in_file/out_file/err_file/out_dir); `:1597: ExtendData = \"\", want ...`; `:1606: ResReq = \"\", want rusage[mem=100]`; `:1637: Command = \"sleep 1\", want the checkpoint's restart command`), GREEN after; with the fix stashed the three fail again. `TestJobHandler_RestartNamedJob` (bug-1898) still passes.\n- Suites: `xce-build.sh apiserver asan xce.build --incremental` (CGO build; apiserver + xcecli Go tests all ok), 2026-10-10; apiserver `CGO_ENABLED=0 go test ./...` all packages ok\n- Live: n/a: handler-level fixture tests only; no live restart of a checkpointed job was run (needs CRIU on a real worker)", "tags": [ "apiserver", "restart", @@ -18731,7 +18731,9 @@ "related_bugs": [ "bug-1936" ], - "fix_commits": [], + "fix_commits": [ + "6a2c69f8" + ], "record": ".wolf/bugs/bug-1968.md", "gitea_issue": 399, "gitea_synced_status": "open" diff --git a/.wolf/bugs/bug-1968.md b/.wolf/bugs/bug-1968.md index 2248eef9..ca97bc0e 100644 --- a/.wolf/bugs/bug-1968.md +++ b/.wolf/bugs/bug-1968.md @@ -1,8 +1,8 @@ --- id: "bug-1968" -status: "open" +status: "fixed" status_updated: "2026-10-10" -status_note: "newly logged" +status_note: "" component: "apiserver" severity: "wrong-result" timestamp: "2026-10-10T11:55:48Z" @@ -11,6 +11,7 @@ occurrences: 1 files: ["src/frontend/apiserver/controllers/jobs.go"] tags: ["apiserver", "restart", "submit"] related: ["bug-1936"] +fix_commits: ["6a2c69f8"] gitea_issue: 399 gitea_synced_status: "open" gitea_synced_hash: "a735a6546a0b" @@ -34,6 +35,8 @@ if v, ok := oldJob["out_dir"].(string); ok { ... } // in_file, err_file, **Impact:** wrong result: a restarted job ignores the asked hosts, begin/term time and I/O paths the user gave, with no error. **Found by:** review during the bug-1936 fix, 2026-10-10. +UPDATE (2026-10-10, during the fix): the same rebuild also lost options the job DOES get. Rows come from `store.StructToOrderMap`, which keeps each field's Go type, so `oldJob["chkpnt_info"].(string)` and `oldJob["extend_data"].(string)` never matched `models.Jsonb`: env/limits (exdata) were dropped on every restart and a checkpointed job never restarted from its checkpoint (`restart_command`, `XCE_JOB_CHECKPOINT_PID`). It also read `oldJob["res_req"]` (the column is `resreq`), dropping the resource requirement, and never copied `exec_home`. + ## Environment - OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 - libc: glibc 2.42-47 (n/a: Go apiserver, CGO_ENABLED=0 tests) @@ -44,21 +47,27 @@ if v, ok := oldJob["out_dir"].(string); ok { ... } // in_file, err_file, ## Reproduction 1. Submit a job, let it finish. -2. `xcecli jobs restart -m ` (or restart a job whose original row has begin_time/out_file). +2. `POST /xce/v1/jobs//restart` with body `{"asked_hosts":""}` (xcecli `jobs restart` has no `-m` flag and sends none), or restart a job whose row has out_file/out_dir set. 3. The restart returns success; the new job is placed without regard to `` (scheduler never receives asked hosts). - Frequency: deterministic (code path) - Evidence: the code quoted in Description; no logs kept. ## Root cause -TODO: required for fixed/partial +`RestartJobByID()` (`src/frontend/apiserver/controllers/jobs.go`) rebuilds the new job by copying fields out of the old job's stored row. Its copy list predates bug-1936: it accepted `asked_hosts` in the request and copied asked hosts, begin/term time, in/out/err files and `out_dir`, none of which any job gets. The list was also never checked against the row's real keys and types: JSONB columns are `models.Jsonb` (not `string`) and the resource requirement key is `resreq` (not `res_req`), so those copies silently did nothing. Pre-existing, origin not bisected. ## Fix -TODO: required for fixed/partial +`controllers/jobs.go`: +- `preCheckJobRestartRequest()` refuses a non-blank `asked_hosts` with 400 ("job restart option asked_hosts not supported"), like `preCheckSubmitOptions()` (bug-1936); the request field stays so the client gets that message, not "unknown field". `processJobRestartRequest()` (which only copied asked hosts) is removed. +- The restart no longer carries over asked hosts, begin/term time, in/out/err files or `out_dir`. +- New `rowJSON()` reads a JSONB row column as `models.Jsonb` or string; used for `chkpnt_info` and `extend_data`, so a checkpointed job restarts with its restart command and checkpoint pid, and every restart keeps the old job's extended data (env, limits). +- Copies `resreq` (was `res_req`) and `exec_home`. ## Verification -- Test: TODO: name + file; RED before the fix, GREEN after -- Suites: TODO: suite results and build profile -- Live: TODO: soak/deploy run, or n/a and why +- Test: `TestJobHandler_RestartRejectsAskedHosts`, `TestJobHandler_RestartCarriesOnlyHonouredOptions`, `TestJobHandler_RestartFromCheckpoint` (`src/frontend/apiserver/controllers/jobs_handler_test.go`): RED before (`jobs_handler_test.go:1537: status = 200, want 400`; `:1592: restarted job kept asked_hosts, which no job gets` (also in_file/out_file/err_file/out_dir); `:1597: ExtendData = "", want ...`; `:1606: ResReq = "", want rusage[mem=100]`; `:1637: Command = "sleep 1", want the checkpoint's restart command`), GREEN after; with the fix stashed the three fail again. `TestJobHandler_RestartNamedJob` (bug-1898) still passes. +- Suites: `xce-build.sh apiserver asan xce.build --incremental` (CGO build; apiserver + xcecli Go tests all ok), 2026-10-10; apiserver `CGO_ENABLED=0 go test ./...` all packages ok +- Live: n/a: handler-level fixture tests only; no live restart of a checkpointed job was run (needs CRIU on a real worker) ## References -- Related: none +- Fix commit: `6a2c69f8` +- Related: bug-1936 (submit-side rejection), bug-1898 (restart identity) +