diff --git a/.wolf/buglog.json b/.wolf/buglog.json index f6e40cb9..b632fa8e 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -16855,6 +16855,78 @@ "related_bugs": [], "fix_commits": [], "record": ".wolf/bugs/bug-1903.md" + }, + { + "id": "bug-1904", + "status": "open", + "status_updated": "2026-10-01", + "status_note": "newly logged", + "component": "scheduler", + "severity": "crash", + "timestamp": "2026-10-01T14:32:17Z", + "last_seen": "2026-10-01T14:32:17Z", + "occurrences": 1, + "error_message": "scheduler crashes on a JOB_UPDATE whose resreq is empty: apply_job_task_count writes through NULL\n\n**Observed:**\n```\n==1580007==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x5c8c80c00867 ...)\n==1580007==The signal is caused by a WRITE memory access.\n #0 apply_job_task_count ../lib/resreq.c:589\n #1 xce_core_push_events ../scheduler/src/core_events.c:1388\n #2 xce_sched_driver_translate_pending_events ../scheduler/src/driver.c:850\n #3 xce_sched_driver_run_normal_mode_round ../scheduler/src/driver.c:1021\n```\nThe scheduler process died (stderr was /dev/null under start-all.sh, so it vanished silently) right after the controller pushed `schedUpdateJob` for a REST-submitted job with no `-R`; every later job stayed PEND. Restarting the scheduler crashed again on the same replayed update.\n\n**Expected:** an update with an empty resource request is treated as ingest treats it (`\"type == any\"`) and the scheduler keeps running.\n**Impact:** any job submitted without a resource request (every REST submit without `-R`) kills the scheduler on its first JOB_UPDATE, and keeps killing it on restart; scheduling stops cluster-wide.\n**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow)", + "file": "src/scheduler/src/core_events.c, src/scheduler/src/schedobj.c", + "root_cause": "Both JOB_UPDATE paths (`core_events.c` `xce_core_push_events`, `schedobj.c` update of an existing job) parsed `updated->submit->resreq` as-is. For `\"\"`, `parse_resreq()` zeroes the container (tag 0 = `XCE_RESREQ_PRIMARY`, `u.primary` NULL), parses nothing and returns success; `apply_job_task_count()` (added there by the bug-1773 fix, 41c3694d) then writes `u.primary->minTasks` through NULL. Ingest (`core_build_job`, `make_sched_job`) already maps an empty string to `\"type == any\"`; the update paths did not.", + "fix": "Both update paths now apply the ingest default: an empty resreq string is parsed as `\"type == any\"`, and `resreq_str` records the string actually parsed.\n\nVerification:\n- Test: `test_job_update_with_empty_resreq_uses_default` in `src/scheduler/testsuite/test_maxtasks_wiring_ingest.c`; RED before the fix (the same ASAN SEGV at resreq.c:589), GREEN after\n- Suites: scheduler meson suite 97/97 OK (asan, src/build.jai)\n- Live: restarted the fixed scheduler on the same deployment: it replayed the update without crashing and every stuck PEND job (plain, arr2[1], arr2[3]) ran to DONE", + "tags": [ + "resreq", + "job-update", + "asan", + "bug-1773-regression" + ], + "related_bugs": [ + "bug-1773" + ], + "fix_commits": [], + "record": ".wolf/bugs/bug-1904.md" + }, + { + "id": "bug-1905", + "status": "open", + "status_updated": "2026-10-01", + "status_note": "newly logged", + "component": "worker", + "severity": "wrong-result", + "timestamp": "2026-10-01T14:32:18Z", + "last_seen": "2026-10-01T14:32:18Z", + "occurrences": 1, + "error_message": "xcecd_virtual ignores kill of a running job: job keeps RUN until its sleep time elapses\n\n**Observed:**\n```\n14:17:47,625 controller_connector.c:1546 Received job control request to send SIGINFO for job ID: 1-0-1790864259-0\n14:17:47,625 controller_connector.c:1610 Job control task handled directly for job ID: 1-0-1790864259-0\n14:27:40,008 xcecd_worker_utils.c:847 Sleep time 600 >= 600 seconds elapsed for worker 1009 (job 1-0-1790864259-0)\nctrl-1: on_job_finish job finish jobid 1-0-1790864259-0 exit code 0\n```\n`xcecli jobs kill` of a RUN job (plain `sleep 600`, and array elements `arr3[2]`, `arr3[3]`) is accepted (apiserver: `overallStatus accepted`, SIGTERM) and reaches the virtual worker, but the job stays RUN and finishes DONE with exit 0 when its sleep time elapses. Killing a PEND job works (EXIT).\n\n**Expected:** the virtual worker ends a killed running job as EXITED with exit code 128+signal on its next poll (the `killed_signal` branch in `xcecd_dbus_manager.c:259-262`).\n**Impact:** kill of running jobs cannot be exercised on virtual-worker deployments (tests, soaks); the real worker path (`ipc_master_send_job_signal`) is not affected by this record.\n**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow)", + "file": "src/worker/src/controller_connector.c, src/worker/src/xcecd_dbus_manager.c", + "root_cause": "TODO: required for fixed/partial", + "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "tags": [ + "virtual-cluster", + "job-control", + "kill" + ], + "related_bugs": [], + "fix_commits": [], + "record": ".wolf/bugs/bug-1905.md" + }, + { + "id": "bug-1906", + "status": "open", + "status_updated": "2026-10-01", + "status_note": "newly logged", + "component": "xcecli", + "severity": "wrong-result", + "timestamp": "2026-10-01T14:32:21Z", + "last_seen": "2026-10-01T14:32:21Z", + "occurrences": 1, + "error_message": "xcecli jobs list drops the selection and its brackets from rejection messages\n\n**Observed:**\n```\n$ xcecli jobs list '1-0-1790863351-0[1,3]'\nfailed to get parameter: [expected , k or a-b]\n$ xcecli jobs kill '1-0-1790863351-0[1,3]'\ninvalid job identity or element selection '1-0-1790863351-0[1,3]': expected , [k] or [a-b]\n```\n`jobs list` passed local selection-parse errors through `request.ExtractErrorMessage`, which strips every `[`/`]` and keeps only the text after the last `\": \"`.\n\n**Expected:** the rejection names the selection as written and keeps the `[k]` syntax, as `jobs kill` does (job-array-one-based-index Requirement 5: a rejection names the rejected selection).\n**Impact:** users of `jobs list` cannot tell which selection was rejected, and the hint shows `k` instead of `[k]`.\n**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow)", + "file": "src/frontend/xcecli/cmd/jobs/list.go", + "root_cause": "`cmd/jobs/list.go` (`RunJobsList`, `runArrayJobStats`) formatted `HandleJobFilter` errors with `request.ExtractErrorMessage`, a sanitizer meant for transport errors: it removes all brackets and keeps only the last `\": \"`-separated part, which drops the quoted selection.", + "fix": "Both sites report the local parse error as written (`failed to get parameter: [%v]`, err), without `ExtractErrorMessage`.\n\nVerification:\n- Test: `TestRunJobsListReportsRejectedSelectionVerbatim` in `src/frontend/xcecli/cmd/jobs/list_test.go`; RED before the fix (6 assertion failures), GREEN after\n- Suites: xcecli `go vet ./...` clean, `go test ./...` all ok (CGO_ENABLED=0); `/xce-build xcecli` asan rc=0\n- Live: rebuilt xcecli against the deployment: `jobs list` rejections now read e.g. `invalid element selection '1-0-1790863351-0[40000]': element numbers must be at most 32767`", + "tags": [ + "job-array", + "error-message", + "selection" + ], + "related_bugs": [], + "fix_commits": [], + "record": ".wolf/bugs/bug-1906.md" } ] } diff --git a/.wolf/bugs/bug-1904.md b/.wolf/bugs/bug-1904.md new file mode 100644 index 00000000..f28ab032 --- /dev/null +++ b/.wolf/bugs/bug-1904.md @@ -0,0 +1,62 @@ +--- +id: "bug-1904" +status: "open" +status_updated: "2026-10-01" +status_note: "newly logged" +component: "scheduler" +severity: "crash" +timestamp: "2026-10-01T14:32:17Z" +last_seen: "2026-10-01T14:32:17Z" +occurrences: 1 +files: ["src/scheduler/src/core_events.c", "src/scheduler/src/schedobj.c"] +tags: ["resreq", "job-update", "asan", "bug-1773-regression"] +related: ["bug-1773"] +--- + +# bug-1904: scheduler crashes on a JOB_UPDATE whose resreq is empty: apply_job_task_count writes through NULL + +## Description +**Observed:** +``` +==1580007==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x5c8c80c00867 ...) +==1580007==The signal is caused by a WRITE memory access. + #0 apply_job_task_count ../lib/resreq.c:589 + #1 xce_core_push_events ../scheduler/src/core_events.c:1388 + #2 xce_sched_driver_translate_pending_events ../scheduler/src/driver.c:850 + #3 xce_sched_driver_run_normal_mode_round ../scheduler/src/driver.c:1021 +``` +The scheduler process died (stderr was /dev/null under start-all.sh, so it vanished silently) right after the controller pushed `schedUpdateJob` for a REST-submitted job with no `-R`; every later job stayed PEND. Restarting the scheduler crashed again on the same replayed update. + +**Expected:** an update with an empty resource request is treated as ingest treats it (`"type == any"`) and the scheduler keeps running. +**Impact:** any job submitted without a resource request (every REST submit without `-R`) kills the scheduler on its first JOB_UPDATE, and keeps killing it on restart; scheduling stops cluster-wide. +**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow) + +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (src/build.jai) +- Code: scheduler-framework @ 6439e585-dirty +- Deployment: 3-node HA (xce-deploy-ha) + scheduler + apiserver + one xcecd_virtual worker, single host + +## Reproduction +1. Deploy with xce-deploy-ha and `start-all.sh`; `xcecli login -u admin -p admin --url http://localhost:8080`. +2. `xcecli jobs submit -J 'arr[1-3]'