diff --git a/.wolf/buglog.json b/.wolf/buglog.json index c4157c30..7d19bec3 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -17815,9 +17815,9 @@ }, { "id": "bug-1936", - "status": "partial", + "status": "fixed", "status_updated": "2026-10-10", - "status_note": "exdata/cwd/home now reach the worker; still not honoured end to end: --cwd (job process spawned with cwd NULL, no cwd in job_assignment_t), exec home (no REST field), nofile_limit (not parsed), -i/-o/-e and out_dir, asked hosts and begin/end time (no submit fields), file stage-in/out (not implemented)", + "status_note": "", "component": "apiserver", "severity": "wrong-result", "timestamp": "2026-10-10T08:14:36Z", @@ -17825,8 +17825,8 @@ "occurrences": 1, "error_message": "apiserver drops cwd, env, run/resource limits, file transfers and I/O paths of REST job submissions\n\n**Observed:** (source reading, `.kiro/reviews/project-gap-review-2026-10-10.md` (P0); to be confirmed by a failing test before the fix)\n`modelJobToSubmit()` (`src/frontend/apiserver/backend/convert.go:323-375`) builds the controller's `job_startinfo_t` with only cmdline, execuser, launch_type and pre/post-exec; it never sets `exdata`, `exechome` or `execcwd`:\n```go\nsi := (*C.job_startinfo_t)(C.calloc(1, C.sizeof_job_startinfo_t))\nsi.cmdline = C.CString(job.Command)\n... // no si.exdata / si.exechome / si.execcwd\n```\nSo `--cwd`, `--env`, `--file` transfers, interactive mode, every `*_limit` flag (incl. `-W` run limit), `-i/-o/-e`, asked hosts and begin/end time are kept only in the apiserver's memory and never reach the controller or the worker.\n\n**Expected:** Every submit option the apiserver accepts reaches the worker in the job's start info (exdata, exec home, cwd), or is rejected as unsupported.\n**Impact:** Silent wrong result: jobs run without the limits, environment, working directory and I/O paths the user asked for (e.g. a `-W` run limit is never enforced).\n**Found by:** review: `.kiro/reviews/project-gap-review-2026-10-10.md` (P0)", "file": "src/frontend/apiserver/backend/convert.go, src/frontend/apiserver/controllers/jobs.go", - "root_cause": "`modelJobToSubmit()` (`backend/convert.go`) filled only cmdline, execuser, launch_type and pre/post-exec of `job_startinfo_t`; the extended data `processJobRequest()` (`controllers/jobs.go`) builds (env, limits, transfers, interactive) and the cwd stayed in apiserver memory. Pre-existing, origin not bisected.", - "fix": "`backend/convert_pure.go`: new pure `submitStartInfo()` sets exdata from `ExtendData`, execcwd (`ExecCWD`, else `CWD`) and exechome; `rawJobToModel` restores `ExtendData`/`CWD`; new `ValidateExdata()` rejects non-object or >16383-byte exdata (the worker would fail the job). `backend/convert.go`: copies/frees the three fields and reads exdata back for jobs and arrays. `controllers/jobs.go`: validates exdata and no longer writes `\"transfer_files\": null`. The controller passes start info through unchanged (WAL submit facts encode the whole JobStartInfo).\n\nVerification:\n- Test: `TestSubmitStartInfo_CarriesExdataCwdAndHome`, `TestSubmitStartInfo_PrefersExecCWD`, `TestRawJobArrayToModels_RestoresExdataAndCwd`, `TestValidateExdata` (`backend/convert_test.go`), `TestJobHandler_SubmitRejectsNonObjectExtendData` (`controllers/jobs_handler_test.go`): RED before (e.g. `convert_test.go:1272: exdata: got \"\", want \"{\\\"limits\\\":{\\\"run_limit\\\":60},...}\"`), GREEN after.\n- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch, 2026-10-10: all C suites Ok 155 / Fail 0, frontend Go unit tests ok, xceadmin tests ok; apiserver `CGO_ENABLED=0 go test ./...` (incl. middleware, router, cmd) ok\n- Live: n/a: unit/fixture-level verification only; no live deployment was run (follow-up)", + "root_cause": "`modelJobToSubmit()` (`backend/convert.go`) filled only cmdline, execuser, launch_type and pre/post-exec of `job_startinfo_t`; the extended data `processJobRequest()` (`controllers/jobs.go`) builds (env, limits, transfers, interactive) and the cwd stayed in apiserver memory. Pre-existing, origin not bisected.\n\nUPDATE (2026-10-10): the remaining gaps: xcecd's job-assignment message to the per-job worker process (`job_assignment_t`, `src/worker/include/uv_ipc.h`) had no cwd or home field, so the job was spawned with no working directory and exechome was never applied; `parse_limits()` (`xcecd_exdata.c`) ignored `nofile_limit`; and the apiserver accepted in/out/err files, out_dir, asked hosts, begin/term time and transfer files, which have no field in the controller's submit data or start info (the worker parses transfer files but never stages them).", + "fix": "`backend/convert_pure.go`: new pure `submitStartInfo()` sets exdata from `ExtendData`, execcwd (`ExecCWD`, else `CWD`) and exechome; `rawJobToModel` restores `ExtendData`/`CWD`; new `ValidateExdata()` rejects non-object or >16383-byte exdata (the worker would fail the job). `backend/convert.go`: copies/frees the three fields and reads exdata back for jobs and arrays. `controllers/jobs.go`: validates exdata and no longer writes `\"transfer_files\": null`. The controller passes start info through unchanged (WAL submit facts encode the whole JobStartInfo).\n\nUPDATE (2026-10-10, `725b1d23`, `edad8c73`), per remaining item:\n- `--cwd`: honoured. `job_assignment_t` gains `cwd`/`home`, filled by new `ipc_fill_job_assignment()` (`uv_ipc.c`, always NUL-terminated); new `uv_proc_enter_job_dirs()` (`uv_proc.c`), called from `worker_job_callback()` (`xcecd_worker.c`), enters the cwd before pre-exec, so pre-exec and the job run there. An unusable cwd falls back to `/tmp` with a WARN (xcecli always sends the client's cwd, so failing would break submissions from directories absent on the worker host). The apiserver rejects a relative or >1023-byte cwd.\n- exec home: honoured. New REST field `exec_home` (`models/jobs.go`, absolute, <=1023 bytes) → start info exechome → the worker sets `HOME` (an explicit `--env HOME=` wins).\n- `nofile_limit`: honoured. `parse_limits()` reads it into `res_limits_t.nofile_limit`; `xcecd_cgroups.c` sets `LimitNOFILE` on the job unit (code-read only: needs D-Bus). xcecli has no flag for it.\n- `in_file`, `out_file`, `err_file`, `out_dir`, `asked_hosts`, `begin_time`, `term_time`, `transfer_files`: rejected. New `preCheckSubmitOptions()` (`controllers/jobs.go`) in `CreateJob` returns 400 \"job submission option(s) ... not supported\" before upload handling or job-id allocation. `-o/-e` were not honoured because the worker's output fetch is locked to its own output path (bug-1508). Swagger regenerated.\nThe same options are still copied and dropped on the job restart path: logged as a separate bug.\n\nVerification:\n- Test: `TestSubmitStartInfo_CarriesExdataCwdAndHome`, `TestSubmitStartInfo_PrefersExecCWD`, `TestRawJobArrayToModels_RestoresExdataAndCwd`, `TestValidateExdata` (`backend/convert_test.go`), `TestJobHandler_SubmitRejectsNonObjectExtendData` (`controllers/jobs_handler_test.go`): RED before (e.g. `convert_test.go:1272: exdata: got \"\", want \"{\\\"limits\\\":{\\\"run_limit\\\":60},...}\"`), GREEN after.\n- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch, 2026-10-10: all C suites Ok 155 / Fail 0, frontend Go unit tests ok, xceadmin tests ok; apiserver `CGO_ENABLED=0 go test ./...` (incl. middleware, router, cmd) ok\n- Second fix: `test_assignment_carries_cwd_and_home`, `..._enters_cwd_and_sets_home`, `..._unusable_cwd_falls_back`, `..._spawned_after_enter_runs_in_cwd` (new `src/worker/testsuite/test_job_dirs.c`; RED e.g. `test_job_dirs.c:111:test_assignment_carries_cwd_and_home:FAIL: Expected '/data/run' Was ''`, `test_job_dirs.c:207:...: the job did not write pwd.out in cwd`), `test_apiserver_nofile_limit_parsed` (`test_exdata_parse.c`; RED `:148: FAIL: Expected 2048 Was 0`), `TestJobHandler_SubmitRejectsUnsupportedOptions` (RED `jobs_handler_test.go:1631: in_file: status = 200, want 400`), `TestJobHandler_SubmitRejectsBadCwdAndExecHome` (RED `:1655: exec_home=home/u: status = 500, want 400`), `TestJobHandler_SubmitCarriesCwdAndExecHome`, `TestJobHandler_SubmitNofileLimitInExdata` (`controllers/jobs_handler_test.go`); GREEN after. `TestJobHandler_SubmitArrayFileNameSubstitution` now checks %J/%I substitution on `cwd` (out/err files are refused).\n- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch (scheduler-framework @ edad8c73), 2026-10-10: all C suites Ok 158 / Fail 0, frontend Go unit tests (apiserver + xcecli) ok; apiserver `CGO_ENABLED=0 go test ./...` (all packages) ok\n- Live: n/a: unit/fixture-level verification only; no live deployment was run (follow-up)", "tags": [ "apiserver", "submit", @@ -17835,7 +17835,9 @@ ], "related_bugs": [], "fix_commits": [ - "d01ced9b" + "d01ced9b", + "725b1d23", + "edad8c73" ], "record": ".wolf/bugs/bug-1936.md", "gitea_issue": 367, @@ -18336,9 +18338,9 @@ }, { "id": "bug-1955", - "status": "partial", + "status": "fixed", "status_updated": "2026-10-10", - "status_note": "covers a signal arriving after this host accepted the runtasks request; a control that reaches a remote host before its runtasks request arrives is still dropped there (needs a controller-side or pre-start record)", + "status_note": "", "component": "worker", "severity": "wrong-result", "timestamp": "2026-10-10T08:46:10Z", @@ -18346,8 +18348,8 @@ "occurrences": 1, "error_message": "xcecd drops a signal for a remote task of a multi-host job that is still starting\n\n**Observed:** (source reading, 2026-10-10; found while fixing bug-1942)\nOnly the head host's job start opens the pending-signal window\n(`pending_signal_start(d->jobid)`, `src/worker/src/controller_connector.c:1062`,\nbug-1908). The runtasks handler for remote tasks (`handle_runtasks()`, `:1147`)\nnever does, so a signal that arrives before the remote task's worker entry is\nregistered is logged and dropped:\n```c\nXCECD_LOG(XCE_LOG_ERROR, \"Failed to find worker for job ID: %s\", jobidStr);\nFREEUP(task_ctx);\nreturn -1; /* controller_connector.c:1606 */\n```\nSince bug-1942 every host of a multi-host job receives job control, so this\nwindow is now reachable.\n\n**Expected:** like the head host (bug-1908), a signal for a remote task that\nis starting is kept and delivered once the task runs.\n**Impact:** wrong result: a kill or stop sent right after a multi-host job\nstarts can miss some of its remote tasks, which keep running.\n**Found by:** review of the bug-1940/1942 fixes, 2026-10-10.", "file": "src/worker/src/controller_connector.c", - "root_cause": "Only the head-host start path opened the pending-signal window (`pending_signal_start`, `controller_connector.c`, bug-1908); `handle_runtasks()` queued remote tasks without it, so a signal arriving before the task's worker entry registered hit \"Failed to find worker\" and was dropped.", - "fix": "`handle_runtasks()` calls `pending_signal_start(jobid)` before `submit_fork_task()` and takes it back if queuing fails; the existing RUNNING transition (`xcecd_worker_utils.c`, bug-1908) delivers the kept signal to the task.\n\nVerification:\n- Test: `test_jobcontrol_signal_for_queued_remote_task_is_kept` (`src/worker/testsuite/test_controller_connector.c`): RED `:1631: FAIL: Expected 0 Was -1. the kill must not be dropped`, GREEN after (32 tests, 0 failures).\n- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch, 2026-10-10: all C suites Ok 155 / Fail 0, frontend Go unit tests ok, xceadmin tests ok; apiserver `CGO_ENABLED=0 go test ./...` (incl. middleware, router, cmd) ok\n- Live: n/a: unit/fixture-level verification only; no live deployment was run (follow-up)", + "root_cause": "Only the head-host start path opened the pending-signal window (`pending_signal_start`, `controller_connector.c`, bug-1908); `handle_runtasks()` queued remote tasks without it, so a signal arriving before the task's worker entry registered hit \"Failed to find worker\" and was dropped.\n\nUPDATE (2026-10-10): the remaining window: job control reaches a remote host from the controller (`job_ctrl()`, `src/controller/job.c`, bug-1942) as soon as the job is dispatched, while that host's runtasks only arrives after the head xcecd starts and asks the controller (`job_start_remote_task()` → `do_runtasks()`, `task.c`, which keeps no per-job state). A control arriving before runtasks found neither a worker entry nor a pending start and was dropped.", + "fix": "`handle_runtasks()` calls `pending_signal_start(jobid)` before `submit_fork_task()` and takes it back if queuing fails; the existing RUNNING transition (`xcecd_worker_utils.c`, bug-1908) delivers the kept signal to the task.\n\nUPDATE (2026-10-10, `81e7187d`): worker-side \"early\" pending signals. `pending_signal.c`: `pending_signal_record_early()` keeps a signal for a job with no start on this host (or records it on a start already pending); `pending_signal_start_task()` (used by `handle_runtasks()`) adopts an unexpired early signal, which the existing RUNNING transition (bug-1908) delivers, so the task exits through the normal finish report (no hung job). Early entries are capped at 256 and expire after 60 s, swept on use under the table mutex (nothing in delaymgr). A head-host start (`pending_signal_start`) discards rather than inherits an early entry. `handle_jobcontrol()` keeps the control early unless the job already finished on this host (`xcecd_finished_job_lookup`). Accepted limits: a control arriving before a rerun's runtasks on the same host within the 60 s finished-job retention is still treated as late; a control for a host the job never starts on lingers up to 60 s.\n\nVerification:\n- Test: `test_jobcontrol_signal_for_queued_remote_task_is_kept` (`src/worker/testsuite/test_controller_connector.c`): RED `:1631: FAIL: Expected 0 Was -1. the kill must not be dropped`, GREEN after. Second fix: `test_jobcontrol_signal_before_remote_runtasks_is_applied` RED `test_controller_connector.c:1695:...:FAIL: Expected 0 Was -1. a kill arriving before runtasks must not be dropped`, GREEN after; also `test_jobcontrol_signal_for_finished_task_is_not_kept`, `test_jobcontrol_signal_for_unknown_job_is_kept_early` (replaces `..._unknown_job_fails`, which asserted the drop), and `test_early_signal_is_adopted_by_task_start`, `test_early_signal_on_pending_start_is_recorded`, `test_stale_early_signal_expires`, `test_early_signals_are_bounded`, `test_head_start_does_not_inherit_early_signal` (`test_pending_signal.c`; RED was a compile error, the functions did not exist).\n- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch (scheduler-framework @ edad8c73), 2026-10-10: all C suites Ok 158 / Fail 0, frontend Go unit tests (apiserver + xcecli) ok; apiserver `CGO_ENABLED=0 go test ./...` (all packages) ok\n- Live: n/a: needs a real multi-host `xcecd` (systemd/dbus), not available in this container. Follow-up: `/xce-test-multihost` topology, kill (and stop) a plain multi-host job right after dispatch; the remote host should log \"signal 9 is kept\" then \"a signal arrived before its tasks\" and the job reach a final state.", "tags": [ "worker", "job-control", @@ -18359,7 +18361,8 @@ "bug-1908" ], "fix_commits": [ - "d3da3140" + "d3da3140", + "81e7187d" ], "record": ".wolf/bugs/bug-1955.md", "gitea_issue": 386, @@ -18678,9 +18681,9 @@ }, { "id": "bug-1967", - "status": "open", + "status": "fixed", "status_updated": "2026-10-10", - "status_note": "newly logged", + "status_note": "", "component": "scheduler", "severity": "wrong-result", "timestamp": "2026-10-10T10:47:20Z", @@ -18688,8 +18691,8 @@ "occurrences": 1, "error_message": "xce_scheduler ignores limit changes in a re-pushed config: seeding stops at the first existing name\n\n**Observed:** (source reading, 2026-10-10; reported by the bug-1964 fix)\n`limit_seed_config_impl()` (`src/scheduler/src/plugins/limit.c`) builds each\nconfigured limit and adds it with `key_add(privdata->limits, limit->name, ...)`,\nreturning at the first failure:\n```c\nrc = key_add(privdata->limits, (void*)limit->name, (void*)limit);\n...\nif (rc < 0) {\n ... /* bug-1685: the kvmgr already freed `limit` */\n return rc;\n}\n```\nOn a later config push (live config upgrade, or a mid-replay CONFIG_PUSH in\nthe simulator since bug-1964) every limit name already exists, so the first\n`key_add` fails with `-EEXIST`: no limit is updated, a limit whose value or\nscope changed keeps its old definition, and limits listed after an existing\none are never added.\n\n**Expected:** a config push replaces the configured limits: changed limits\ntake their new definition, new ones are added, removed ones disappear\n(their usage instances handled consistently).\n**Impact:** wrong result: limit changes made by a live config upgrade do not\ntake effect until the scheduler restarts; live and simulator agree (both\nwrong).\n**Found by:** fix of bug-1964 (sim limit config push), 2026-10-10.", "file": "src/scheduler/src/plugins/limit.c", - "root_cause": "TODO: required for fixed/partial", - "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "root_cause": "`limit_seed_config_impl()` (`src/scheduler/src/plugins/limit.c`) added each pushed limit to the existing `privdata->limits` table and returned at the first `key_add()` failure; kvmgr has no delete, so on any re-push the first already-present name failed with `-EEXIST`: old definitions stayed, later new limits were never added and removed ones never dropped. Live config upgrade (`limit_update_config`) and the sim's CONFIG_PUSH (`plugin_config_update_core`, bug-1964) share this path. Pre-existing since limits were seeded from config.", + "fix": "New `src/scheduler/src/plugins/limit_config.c` `limit_replace_config()` (private data type moved to `limit_internal.h`; `limit.c` was at its line cap):\n- builds the whole new limit set in a fresh table, all-or-nothing: a build failure or duplicate name discards it and keeps the previous limits and instances (bug-1685 ownership kept: a `key_add`-rejected limit is not freed twice; a `build_limit` failure now frees the half-built limit);\n- carries usage instances across: a limit kept under the same name with the same per-user/queue/project/host scoping, resource and percent flag keeps its instances, rebound to the new definition (usage lives in the resource manager's vres, untouched); instances of removed or re-scoped limits are dropped and re-created on the next match;\n- swaps the tables, then frees old instances, then old limits.\nJobs' `glimits`/`hlimits` may briefly hold stale instance pointers; they are rebuilt by `limit_job_election()` before any read (pushes apply between rounds and bump `configver`), documented in the file header. Limits/instances are not in the scheduler snapshot (it reseeds from config), so no save/restore change. Behaviour change: a config containing one bad limit now yields no new limits (the previous set stays) instead of keeping the limits listed before the bad one.\n\nVerification:\n- Test: `test_repush_changed_value_takes_effect`, `test_repush_adds_new_limit_after_existing`, `test_repush_removes_dropped_limit`, `test_repush_rebinds_kept_and_drops_removed_instances`, `test_repush_scope_change_drops_instance`, `test_repush_failure_keeps_previous_set` (`src/scheduler/testsuite/test_limit_plugin.c`): RED `test_limit_plugin.c:595:test_repush_changed_value_takes_effect:FAIL: Expected 0 Was -17` (same at :614, :636, :669, :702), GREEN after. `test_repl_limit_table.c` now pushes limitA+limitB in one config (it relied on the bug).\n- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch (scheduler-framework @ edad8c73), 2026-10-10: all C suites Ok 158 / Fail 0, frontend Go unit tests (apiserver + xcecli) ok; apiserver `CGO_ENABLED=0 go test ./...` (all packages) ok\n- Live: n/a: unit-level only; no live config-upgrade run (follow-up: push a changed limit to a running scheduler and check REPL `info limit all`)", "tags": [ "scheduler", "limit", @@ -18699,10 +18702,37 @@ "bug-1964", "bug-1685" ], - "fix_commits": [], + "fix_commits": [ + "8faf5961" + ], "record": ".wolf/bugs/bug-1967.md", "gitea_issue": 398, "gitea_synced_status": "open" + }, + { + "id": "bug-1968", + "status": "open", + "status_updated": "2026-10-10", + "status_note": "newly logged", + "component": "apiserver", + "severity": "wrong-result", + "timestamp": "2026-10-10T11:55:48Z", + "last_seen": "2026-10-10T11:55:48Z", + "occurrences": 1, + "error_message": "apiserver job restart copies asked hosts, begin/term time and I/O paths it can never pass on\n\n**Observed:** (source reading, 2026-10-10; found while fixing bug-1936)\n`RestartJobByID()` (`src/frontend/apiserver/controllers/jobs.go:832`) builds the new job from the request plus the old job's row, and copies options that bug-1936 now refuses at submit because no field carries them to the controller or worker:\n```go\nif newJob.AskedHosts == \"\" {\n if v, ok := oldJob[\"asked_hosts\"].(string); ok { newJob.AskedHosts = v }\n}\nif v, ok := oldJob[\"begin_time\"].(time.Time); ok { ... } // also term_time,\nif v, ok := oldJob[\"out_dir\"].(string); ok { ... } // in_file, err_file, out_file\n```\n`preCheckJobRestartRequest()` (`:1350`) validates `-m` asked hosts but does not refuse them the way `preCheckSubmitOptions()` (`:1307`, bug-1936) does for submit, so `xcecli jobs restart -m host ...` returns success and the host request is silently dropped.\n\n**Expected:** restart, like submit, either passes each option on to the job or rejects it with 400 (`preCheckSubmitOptions`-style); options copied from the old job that cannot be honoured are not silently kept in apiserver memory only.\n**Impact:** wrong result: a restarted job ignores the asked hosts, begin/term time and I/O paths the user gave, with no error.\n**Found by:** review during the bug-1936 fix, 2026-10-10.", + "file": "src/frontend/apiserver/controllers/jobs.go", + "root_cause": "TODO: required for fixed/partial", + "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "tags": [ + "apiserver", + "restart", + "submit" + ], + "related_bugs": [ + "bug-1936" + ], + "fix_commits": [], + "record": ".wolf/bugs/bug-1968.md" } ] } diff --git a/.wolf/bugs/bug-1936.md b/.wolf/bugs/bug-1936.md index 58985847..9912a6ae 100644 --- a/.wolf/bugs/bug-1936.md +++ b/.wolf/bugs/bug-1936.md @@ -1,8 +1,8 @@ --- id: "bug-1936" -status: "partial" +status: "fixed" status_updated: "2026-10-10" -status_note: "exdata/cwd/home now reach the worker; still not honoured end to end: --cwd (job process spawned with cwd NULL, no cwd in job_assignment_t), exec home (no REST field), nofile_limit (not parsed), -i/-o/-e and out_dir, asked hosts and begin/end time (no submit fields), file stage-in/out (not implemented)" +status_note: "" component: "apiserver" severity: "wrong-result" timestamp: "2026-10-10T08:14:36Z" @@ -11,7 +11,7 @@ occurrences: 1 files: ["src/frontend/apiserver/backend/convert.go", "src/frontend/apiserver/controllers/jobs.go"] tags: ["apiserver", "submit", "exdata", "limits"] related: [] -fix_commits: ["d01ced9b"] +fix_commits: ["d01ced9b", "725b1d23", "edad8c73"] gitea_issue: 367 gitea_synced_status: "partial" gitea_synced_hash: "658095c85aa2" @@ -48,14 +48,25 @@ So `--cwd`, `--env`, `--file` transfers, interactive mode, every `*_limit` flag ## Root cause `modelJobToSubmit()` (`backend/convert.go`) filled only cmdline, execuser, launch_type and pre/post-exec of `job_startinfo_t`; the extended data `processJobRequest()` (`controllers/jobs.go`) builds (env, limits, transfers, interactive) and the cwd stayed in apiserver memory. Pre-existing, origin not bisected. +UPDATE (2026-10-10): the remaining gaps: xcecd's job-assignment message to the per-job worker process (`job_assignment_t`, `src/worker/include/uv_ipc.h`) had no cwd or home field, so the job was spawned with no working directory and exechome was never applied; `parse_limits()` (`xcecd_exdata.c`) ignored `nofile_limit`; and the apiserver accepted in/out/err files, out_dir, asked hosts, begin/term time and transfer files, which have no field in the controller's submit data or start info (the worker parses transfer files but never stages them). + ## Fix `backend/convert_pure.go`: new pure `submitStartInfo()` sets exdata from `ExtendData`, execcwd (`ExecCWD`, else `CWD`) and exechome; `rawJobToModel` restores `ExtendData`/`CWD`; new `ValidateExdata()` rejects non-object or >16383-byte exdata (the worker would fail the job). `backend/convert.go`: copies/frees the three fields and reads exdata back for jobs and arrays. `controllers/jobs.go`: validates exdata and no longer writes `"transfer_files": null`. The controller passes start info through unchanged (WAL submit facts encode the whole JobStartInfo). +UPDATE (2026-10-10, `725b1d23`, `edad8c73`), per remaining item: +- `--cwd`: honoured. `job_assignment_t` gains `cwd`/`home`, filled by new `ipc_fill_job_assignment()` (`uv_ipc.c`, always NUL-terminated); new `uv_proc_enter_job_dirs()` (`uv_proc.c`), called from `worker_job_callback()` (`xcecd_worker.c`), enters the cwd before pre-exec, so pre-exec and the job run there. An unusable cwd falls back to `/tmp` with a WARN (xcecli always sends the client's cwd, so failing would break submissions from directories absent on the worker host). The apiserver rejects a relative or >1023-byte cwd. +- exec home: honoured. New REST field `exec_home` (`models/jobs.go`, absolute, <=1023 bytes) → start info exechome → the worker sets `HOME` (an explicit `--env HOME=` wins). +- `nofile_limit`: honoured. `parse_limits()` reads it into `res_limits_t.nofile_limit`; `xcecd_cgroups.c` sets `LimitNOFILE` on the job unit (code-read only: needs D-Bus). xcecli has no flag for it. +- `in_file`, `out_file`, `err_file`, `out_dir`, `asked_hosts`, `begin_time`, `term_time`, `transfer_files`: rejected. New `preCheckSubmitOptions()` (`controllers/jobs.go`) in `CreateJob` returns 400 "job submission option(s) ... not supported" before upload handling or job-id allocation. `-o/-e` were not honoured because the worker's output fetch is locked to its own output path (bug-1508). Swagger regenerated. +The same options are still copied and dropped on the job restart path: logged as a separate bug. + ## Verification - Test: `TestSubmitStartInfo_CarriesExdataCwdAndHome`, `TestSubmitStartInfo_PrefersExecCWD`, `TestRawJobArrayToModels_RestoresExdataAndCwd`, `TestValidateExdata` (`backend/convert_test.go`), `TestJobHandler_SubmitRejectsNonObjectExtendData` (`controllers/jobs_handler_test.go`): RED before (e.g. `convert_test.go:1272: exdata: got "", want "{\"limits\":{\"run_limit\":60},...}"`), GREEN after. - Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch, 2026-10-10: all C suites Ok 155 / Fail 0, frontend Go unit tests ok, xceadmin tests ok; apiserver `CGO_ENABLED=0 go test ./...` (incl. middleware, router, cmd) ok +- Second fix: `test_assignment_carries_cwd_and_home`, `..._enters_cwd_and_sets_home`, `..._unusable_cwd_falls_back`, `..._spawned_after_enter_runs_in_cwd` (new `src/worker/testsuite/test_job_dirs.c`; RED e.g. `test_job_dirs.c:111:test_assignment_carries_cwd_and_home:FAIL: Expected '/data/run' Was ''`, `test_job_dirs.c:207:...: the job did not write pwd.out in cwd`), `test_apiserver_nofile_limit_parsed` (`test_exdata_parse.c`; RED `:148: FAIL: Expected 2048 Was 0`), `TestJobHandler_SubmitRejectsUnsupportedOptions` (RED `jobs_handler_test.go:1631: in_file: status = 200, want 400`), `TestJobHandler_SubmitRejectsBadCwdAndExecHome` (RED `:1655: exec_home=home/u: status = 500, want 400`), `TestJobHandler_SubmitCarriesCwdAndExecHome`, `TestJobHandler_SubmitNofileLimitInExdata` (`controllers/jobs_handler_test.go`); GREEN after. `TestJobHandler_SubmitArrayFileNameSubstitution` now checks %J/%I substitution on `cwd` (out/err files are refused). +- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch (scheduler-framework @ edad8c73), 2026-10-10: all C suites Ok 158 / Fail 0, frontend Go unit tests (apiserver + xcecli) ok; apiserver `CGO_ENABLED=0 go test ./...` (all packages) ok - Live: n/a: unit/fixture-level verification only; no live deployment was run (follow-up) ## References -- Fix commit: `d01ced9b` (on `scheduler-framework`; worker-branch original in the merge) +- Fix commits: `d01ced9b` (first fix), `725b1d23`, `edad8c73` (worktree originals `3e571f25`, `200127bb`) - Review: `.kiro/reviews/project-gap-review-2026-10-10.md` diff --git a/.wolf/bugs/bug-1955.md b/.wolf/bugs/bug-1955.md index 417c4849..198a58bf 100644 --- a/.wolf/bugs/bug-1955.md +++ b/.wolf/bugs/bug-1955.md @@ -1,8 +1,8 @@ --- id: "bug-1955" -status: "partial" +status: "fixed" status_updated: "2026-10-10" -status_note: "covers a signal arriving after this host accepted the runtasks request; a control that reaches a remote host before its runtasks request arrives is still dropped there (needs a controller-side or pre-start record)" +status_note: "" component: "worker" severity: "wrong-result" timestamp: "2026-10-10T08:46:10Z" @@ -11,7 +11,7 @@ occurrences: 1 files: ["src/worker/src/controller_connector.c"] tags: ["worker", "job-control", "parallel", "signal"] related: ["bug-1942", "bug-1908"] -fix_commits: ["d3da3140"] +fix_commits: ["d3da3140", "81e7187d"] gitea_issue: 386 gitea_synced_status: "partial" gitea_synced_hash: "095558c6c67a" @@ -57,14 +57,19 @@ starts can miss some of its remote tasks, which keep running. ## Root cause Only the head-host start path opened the pending-signal window (`pending_signal_start`, `controller_connector.c`, bug-1908); `handle_runtasks()` queued remote tasks without it, so a signal arriving before the task's worker entry registered hit "Failed to find worker" and was dropped. +UPDATE (2026-10-10): the remaining window: job control reaches a remote host from the controller (`job_ctrl()`, `src/controller/job.c`, bug-1942) as soon as the job is dispatched, while that host's runtasks only arrives after the head xcecd starts and asks the controller (`job_start_remote_task()` → `do_runtasks()`, `task.c`, which keeps no per-job state). A control arriving before runtasks found neither a worker entry nor a pending start and was dropped. + ## Fix `handle_runtasks()` calls `pending_signal_start(jobid)` before `submit_fork_task()` and takes it back if queuing fails; the existing RUNNING transition (`xcecd_worker_utils.c`, bug-1908) delivers the kept signal to the task. +UPDATE (2026-10-10, `81e7187d`): worker-side "early" pending signals. `pending_signal.c`: `pending_signal_record_early()` keeps a signal for a job with no start on this host (or records it on a start already pending); `pending_signal_start_task()` (used by `handle_runtasks()`) adopts an unexpired early signal, which the existing RUNNING transition (bug-1908) delivers, so the task exits through the normal finish report (no hung job). Early entries are capped at 256 and expire after 60 s, swept on use under the table mutex (nothing in delaymgr). A head-host start (`pending_signal_start`) discards rather than inherits an early entry. `handle_jobcontrol()` keeps the control early unless the job already finished on this host (`xcecd_finished_job_lookup`). Accepted limits: a control arriving before a rerun's runtasks on the same host within the 60 s finished-job retention is still treated as late; a control for a host the job never starts on lingers up to 60 s. + ## Verification -- Test: `test_jobcontrol_signal_for_queued_remote_task_is_kept` (`src/worker/testsuite/test_controller_connector.c`): RED `:1631: FAIL: Expected 0 Was -1. the kill must not be dropped`, GREEN after (32 tests, 0 failures). -- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch, 2026-10-10: all C suites Ok 155 / Fail 0, frontend Go unit tests ok, xceadmin tests ok; apiserver `CGO_ENABLED=0 go test ./...` (incl. middleware, router, cmd) ok -- Live: n/a: unit/fixture-level verification only; no live deployment was run (follow-up) +- Test: `test_jobcontrol_signal_for_queued_remote_task_is_kept` (`src/worker/testsuite/test_controller_connector.c`): RED `:1631: FAIL: Expected 0 Was -1. the kill must not be dropped`, GREEN after. Second fix: `test_jobcontrol_signal_before_remote_runtasks_is_applied` RED `test_controller_connector.c:1695:...:FAIL: Expected 0 Was -1. a kill arriving before runtasks must not be dropped`, GREEN after; also `test_jobcontrol_signal_for_finished_task_is_not_kept`, `test_jobcontrol_signal_for_unknown_job_is_kept_early` (replaces `..._unknown_job_fails`, which asserted the drop), and `test_early_signal_is_adopted_by_task_start`, `test_early_signal_on_pending_start_is_recorded`, `test_stale_early_signal_expires`, `test_early_signals_are_bounded`, `test_head_start_does_not_inherit_early_signal` (`test_pending_signal.c`; RED was a compile error, the functions did not exist). +- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch (scheduler-framework @ edad8c73), 2026-10-10: all C suites Ok 158 / Fail 0, frontend Go unit tests (apiserver + xcecli) ok; apiserver `CGO_ENABLED=0 go test ./...` (all packages) ok +- Live: n/a: needs a real multi-host `xcecd` (systemd/dbus), not available in this container. Follow-up: `/xce-test-multihost` topology, kill (and stop) a plain multi-host job right after dispatch; the remote host should log "signal 9 is kept" then "a signal arrived before its tasks" and the job reach a final state. ## References -- Fix commit: `d3da3140` (on `scheduler-framework`; worker-branch original in the merge) +- Fix commits: `d3da3140`, `81e7187d` (worktree original `a66227a5`) - Review: `.kiro/reviews/project-gap-review-2026-10-10.md` + diff --git a/.wolf/bugs/bug-1967.md b/.wolf/bugs/bug-1967.md index 42ef6b41..53ed84b2 100644 --- a/.wolf/bugs/bug-1967.md +++ b/.wolf/bugs/bug-1967.md @@ -1,8 +1,8 @@ --- id: "bug-1967" -status: "open" +status: "fixed" status_updated: "2026-10-10" -status_note: "newly logged" +status_note: "" component: "scheduler" severity: "wrong-result" timestamp: "2026-10-10T10:47:20Z" @@ -11,6 +11,7 @@ occurrences: 1 files: ["src/scheduler/src/plugins/limit.c"] tags: ["scheduler", "limit", "config"] related: ["bug-1964", "bug-1685"] +fix_commits: ["8faf5961"] gitea_issue: 398 gitea_synced_status: "open" gitea_synced_hash: "1b09440ec632" @@ -59,15 +60,21 @@ wrong). - Frequency: deterministic (code path) - Evidence: the code quoted in Description; no logs kept. ## Root cause -TODO: required for fixed/partial +`limit_seed_config_impl()` (`src/scheduler/src/plugins/limit.c`) added each pushed limit to the existing `privdata->limits` table and returned at the first `key_add()` failure; kvmgr has no delete, so on any re-push the first already-present name failed with `-EEXIST`: old definitions stayed, later new limits were never added and removed ones never dropped. Live config upgrade (`limit_update_config`) and the sim's CONFIG_PUSH (`plugin_config_update_core`, bug-1964) share this path. Pre-existing since limits were seeded from config. ## Fix -TODO: required for fixed/partial +New `src/scheduler/src/plugins/limit_config.c` `limit_replace_config()` (private data type moved to `limit_internal.h`; `limit.c` was at its line cap): +- builds the whole new limit set in a fresh table, all-or-nothing: a build failure or duplicate name discards it and keeps the previous limits and instances (bug-1685 ownership kept: a `key_add`-rejected limit is not freed twice; a `build_limit` failure now frees the half-built limit); +- carries usage instances across: a limit kept under the same name with the same per-user/queue/project/host scoping, resource and percent flag keeps its instances, rebound to the new definition (usage lives in the resource manager's vres, untouched); instances of removed or re-scoped limits are dropped and re-created on the next match; +- swaps the tables, then frees old instances, then old limits. +Jobs' `glimits`/`hlimits` may briefly hold stale instance pointers; they are rebuilt by `limit_job_election()` before any read (pushes apply between rounds and bump `configver`), documented in the file header. Limits/instances are not in the scheduler snapshot (it reseeds from config), so no save/restore change. Behaviour change: a config containing one bad limit now yields no new limits (the previous set stays) instead of keeping the limits listed before the bad one. ## Verification -- Test: TODO: name + file; RED before the fix, GREEN after -- Suites: TODO: suite results and build profile -- Live: TODO: soak/deploy run, or n/a and why +- Test: `test_repush_changed_value_takes_effect`, `test_repush_adds_new_limit_after_existing`, `test_repush_removes_dropped_limit`, `test_repush_rebinds_kept_and_drops_removed_instances`, `test_repush_scope_change_drops_instance`, `test_repush_failure_keeps_previous_set` (`src/scheduler/testsuite/test_limit_plugin.c`): RED `test_limit_plugin.c:595:test_repush_changed_value_takes_effect:FAIL: Expected 0 Was -17` (same at :614, :636, :669, :702), GREEN after. `test_repl_limit_table.c` now pushes limitA+limitB in one config (it relied on the bug). +- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch (scheduler-framework @ edad8c73), 2026-10-10: all C suites Ok 158 / Fail 0, frontend Go unit tests (apiserver + xcecli) ok; apiserver `CGO_ENABLED=0 go test ./...` (all packages) ok +- Live: n/a: unit-level only; no live config-upgrade run (follow-up: push a changed limit to a running scheduler and check REPL `info limit all`) ## References -- Related: none +- Fix commit: `8faf5961` (worktree original `9bd927d8`) +- Related: bug-1964, bug-1685 + diff --git a/.wolf/bugs/bug-1968.md b/.wolf/bugs/bug-1968.md new file mode 100644 index 00000000..9e0d020a --- /dev/null +++ b/.wolf/bugs/bug-1968.md @@ -0,0 +1,61 @@ +--- +id: "bug-1968" +status: "open" +status_updated: "2026-10-10" +status_note: "newly logged" +component: "apiserver" +severity: "wrong-result" +timestamp: "2026-10-10T11:55:48Z" +last_seen: "2026-10-10T11:55:48Z" +occurrences: 1 +files: ["src/frontend/apiserver/controllers/jobs.go"] +tags: ["apiserver", "restart", "submit"] +related: ["bug-1936"] +--- + +# bug-1968: apiserver job restart copies asked hosts, begin/term time and I/O paths it can never pass on + +## Description +**Observed:** (source reading, 2026-10-10; found while fixing bug-1936) +`RestartJobByID()` (`src/frontend/apiserver/controllers/jobs.go:832`) builds the new job from the request plus the old job's row, and copies options that bug-1936 now refuses at submit because no field carries them to the controller or worker: +```go +if newJob.AskedHosts == "" { + if v, ok := oldJob["asked_hosts"].(string); ok { newJob.AskedHosts = v } +} +if v, ok := oldJob["begin_time"].(time.Time); ok { ... } // also term_time, +if v, ok := oldJob["out_dir"].(string); ok { ... } // in_file, err_file, out_file +``` +`preCheckJobRestartRequest()` (`:1350`) validates `-m` asked hosts but does not refuse them the way `preCheckSubmitOptions()` (`:1307`, bug-1936) does for submit, so `xcecli jobs restart -m host ...` returns success and the host request is silently dropped. + +**Expected:** restart, like submit, either passes each option on to the job or rejects it with 400 (`preCheckSubmitOptions`-style); options copied from the old job that cannot be honoured are not silently kept in apiserver memory only. +**Impact:** wrong result: a restarted job ignores the asked hosts, begin/term time and I/O paths the user gave, with no error. +**Found by:** review during the bug-1936 fix, 2026-10-10. + +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (n/a: Go apiserver, CGO_ENABLED=0 tests) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (xce.build) +- Code: scheduler-framework @ edad8c73-dirty +- Deployment: any (found by source reading) + +## Reproduction +1. Submit a job, let it finish. +2. `xcecli jobs restart -m ` (or restart a job whose original row has begin_time/out_file). +3. The restart returns success; the new job is placed without regard to `` (scheduler never receives asked hosts). +- Frequency: deterministic (code path) +- Evidence: the code quoted in Description; no logs kept. + +## Root cause +TODO: required for fixed/partial + +## Fix +TODO: required for fixed/partial + +## Verification +- Test: TODO: name + file; RED before the fix, GREEN after +- Suites: TODO: suite results and build profile +- Live: TODO: soak/deploy run, or n/a and why + +## References +- Related: none