fix: scheduler empty-resreq JOB_UPDATE crash (bug-1904); xcecli jobs list rejection text (bug-1906)
bug-1904: both scheduler JOB_UPDATE paths parsed "" as-is:
parse_resreq() returns success with a PRIMARY container whose body is
NULL, and apply_job_task_count() wrote through it. Use ingest's default
("type == any") instead.
bug-1906: xcecli jobs list sent local selection-parse errors through
ExtractErrorMessage, which strips brackets and the quoted selection;
report them as written.
Also logs bug-1905 (xcecd_virtual ignores kill of a running job). All
three found by job-array-one-based-index task 10.2's live REST flow.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
6439e5852f
commit
6e631caa22
GPG key ID:
80533536B0EB651C
9 changed files with 392 additions and 12 deletions
|
|
@ -16855,6 +16855,78 @@
|
|||
"related_bugs": [],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1903.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1904",
|
||||
"status": "open",
|
||||
"status_updated": "2026-10-01",
|
||||
"status_note": "newly logged",
|
||||
"component": "scheduler",
|
||||
"severity": "crash",
|
||||
"timestamp": "2026-10-01T14:32:17Z",
|
||||
"last_seen": "2026-10-01T14:32:17Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "scheduler crashes on a JOB_UPDATE whose resreq is empty: apply_job_task_count writes through NULL\n\n**Observed:**\n```\n==1580007==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x5c8c80c00867 ...)\n==1580007==The signal is caused by a WRITE memory access.\n #0 apply_job_task_count ../lib/resreq.c:589\n #1 xce_core_push_events ../scheduler/src/core_events.c:1388\n #2 xce_sched_driver_translate_pending_events ../scheduler/src/driver.c:850\n #3 xce_sched_driver_run_normal_mode_round ../scheduler/src/driver.c:1021\n```\nThe scheduler process died (stderr was /dev/null under start-all.sh, so it vanished silently) right after the controller pushed `schedUpdateJob` for a REST-submitted job with no `-R`; every later job stayed PEND. Restarting the scheduler crashed again on the same replayed update.\n\n**Expected:** an update with an empty resource request is treated as ingest treats it (`\"type == any\"`) and the scheduler keeps running.\n**Impact:** any job submitted without a resource request (every REST submit without `-R`) kills the scheduler on its first JOB_UPDATE, and keeps killing it on restart; scheduling stops cluster-wide.\n**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow)",
|
||||
"file": "src/scheduler/src/core_events.c, src/scheduler/src/schedobj.c",
|
||||
"root_cause": "Both JOB_UPDATE paths (`core_events.c` `xce_core_push_events`, `schedobj.c` update of an existing job) parsed `updated->submit->resreq` as-is. For `\"\"`, `parse_resreq()` zeroes the container (tag 0 = `XCE_RESREQ_PRIMARY`, `u.primary` NULL), parses nothing and returns success; `apply_job_task_count()` (added there by the bug-1773 fix, 41c3694d) then writes `u.primary->minTasks` through NULL. Ingest (`core_build_job`, `make_sched_job`) already maps an empty string to `\"type == any\"`; the update paths did not.",
|
||||
"fix": "Both update paths now apply the ingest default: an empty resreq string is parsed as `\"type == any\"`, and `resreq_str` records the string actually parsed.\n\nVerification:\n- Test: `test_job_update_with_empty_resreq_uses_default` in `src/scheduler/testsuite/test_maxtasks_wiring_ingest.c`; RED before the fix (the same ASAN SEGV at resreq.c:589), GREEN after\n- Suites: scheduler meson suite 97/97 OK (asan, src/build.jai)\n- Live: restarted the fixed scheduler on the same deployment: it replayed the update without crashing and every stuck PEND job (plain, arr2[1], arr2[3]) ran to DONE",
|
||||
"tags": [
|
||||
"resreq",
|
||||
"job-update",
|
||||
"asan",
|
||||
"bug-1773-regression"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1773"
|
||||
],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1904.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1905",
|
||||
"status": "open",
|
||||
"status_updated": "2026-10-01",
|
||||
"status_note": "newly logged",
|
||||
"component": "worker",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-10-01T14:32:18Z",
|
||||
"last_seen": "2026-10-01T14:32:18Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "xcecd_virtual ignores kill of a running job: job keeps RUN until its sleep time elapses\n\n**Observed:**\n```\n14:17:47,625 controller_connector.c:1546 Received job control request to send SIGINFO for job ID: 1-0-1790864259-0\n14:17:47,625 controller_connector.c:1610 Job control task handled directly for job ID: 1-0-1790864259-0\n14:27:40,008 xcecd_worker_utils.c:847 Sleep time 600 >= 600 seconds elapsed for worker 1009 (job 1-0-1790864259-0)\nctrl-1: on_job_finish job finish jobid 1-0-1790864259-0 exit code 0\n```\n`xcecli jobs kill` of a RUN job (plain `sleep 600`, and array elements `arr3[2]`, `arr3[3]`) is accepted (apiserver: `overallStatus accepted`, SIGTERM) and reaches the virtual worker, but the job stays RUN and finishes DONE with exit 0 when its sleep time elapses. Killing a PEND job works (EXIT).\n\n**Expected:** the virtual worker ends a killed running job as EXITED with exit code 128+signal on its next poll (the `killed_signal` branch in `xcecd_dbus_manager.c:259-262`).\n**Impact:** kill of running jobs cannot be exercised on virtual-worker deployments (tests, soaks); the real worker path (`ipc_master_send_job_signal`) is not affected by this record.\n**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow)",
|
||||
"file": "src/worker/src/controller_connector.c, src/worker/src/xcecd_dbus_manager.c",
|
||||
"root_cause": "TODO: required for fixed/partial",
|
||||
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
|
||||
"tags": [
|
||||
"virtual-cluster",
|
||||
"job-control",
|
||||
"kill"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1905.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1906",
|
||||
"status": "open",
|
||||
"status_updated": "2026-10-01",
|
||||
"status_note": "newly logged",
|
||||
"component": "xcecli",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-10-01T14:32:21Z",
|
||||
"last_seen": "2026-10-01T14:32:21Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "xcecli jobs list drops the selection and its brackets from rejection messages\n\n**Observed:**\n```\n$ xcecli jobs list '1-0-1790863351-0[1,3]'\nfailed to get parameter: [expected <job-id>, <array-id>k or <array-id>a-b]\n$ xcecli jobs kill '1-0-1790863351-0[1,3]'\ninvalid job identity or element selection '1-0-1790863351-0[1,3]': expected <job-id>, <array-id>[k] or <array-id>[a-b]\n```\n`jobs list` passed local selection-parse errors through `request.ExtractErrorMessage`, which strips every `[`/`]` and keeps only the text after the last `\": \"`.\n\n**Expected:** the rejection names the selection as written and keeps the `[k]` syntax, as `jobs kill` does (job-array-one-based-index Requirement 5: a rejection names the rejected selection).\n**Impact:** users of `jobs list` cannot tell which selection was rejected, and the hint shows `<array-id>k` instead of `<array-id>[k]`.\n**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow)",
|
||||
"file": "src/frontend/xcecli/cmd/jobs/list.go",
|
||||
"root_cause": "`cmd/jobs/list.go` (`RunJobsList`, `runArrayJobStats`) formatted `HandleJobFilter` errors with `request.ExtractErrorMessage`, a sanitizer meant for transport errors: it removes all brackets and keeps only the last `\": \"`-separated part, which drops the quoted selection.",
|
||||
"fix": "Both sites report the local parse error as written (`failed to get parameter: [%v]`, err), without `ExtractErrorMessage`.\n\nVerification:\n- Test: `TestRunJobsListReportsRejectedSelectionVerbatim` in `src/frontend/xcecli/cmd/jobs/list_test.go`; RED before the fix (6 assertion failures), GREEN after\n- Suites: xcecli `go vet ./...` clean, `go test ./...` all ok (CGO_ENABLED=0); `/xce-build xcecli` asan rc=0\n- Live: rebuilt xcecli against the deployment: `jobs list` rejections now read e.g. `invalid element selection '1-0-1790863351-0[40000]': element numbers must be at most 32767`",
|
||||
"tags": [
|
||||
"job-array",
|
||||
"error-message",
|
||||
"selection"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1906.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
62
.wolf/bugs/bug-1904.md
Normal file
62
.wolf/bugs/bug-1904.md
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
---
|
||||
id: "bug-1904"
|
||||
status: "open"
|
||||
status_updated: "2026-10-01"
|
||||
status_note: "newly logged"
|
||||
component: "scheduler"
|
||||
severity: "crash"
|
||||
timestamp: "2026-10-01T14:32:17Z"
|
||||
last_seen: "2026-10-01T14:32:17Z"
|
||||
occurrences: 1
|
||||
files: ["src/scheduler/src/core_events.c", "src/scheduler/src/schedobj.c"]
|
||||
tags: ["resreq", "job-update", "asan", "bug-1773-regression"]
|
||||
related: ["bug-1773"]
|
||||
---
|
||||
|
||||
# bug-1904: scheduler crashes on a JOB_UPDATE whose resreq is empty: apply_job_task_count writes through NULL
|
||||
|
||||
## Description
|
||||
**Observed:**
|
||||
```
|
||||
==1580007==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x5c8c80c00867 ...)
|
||||
==1580007==The signal is caused by a WRITE memory access.
|
||||
#0 apply_job_task_count ../lib/resreq.c:589
|
||||
#1 xce_core_push_events ../scheduler/src/core_events.c:1388
|
||||
#2 xce_sched_driver_translate_pending_events ../scheduler/src/driver.c:850
|
||||
#3 xce_sched_driver_run_normal_mode_round ../scheduler/src/driver.c:1021
|
||||
```
|
||||
The scheduler process died (stderr was /dev/null under start-all.sh, so it vanished silently) right after the controller pushed `schedUpdateJob` for a REST-submitted job with no `-R`; every later job stayed PEND. Restarting the scheduler crashed again on the same replayed update.
|
||||
|
||||
**Expected:** an update with an empty resource request is treated as ingest treats it (`"type == any"`) and the scheduler keeps running.
|
||||
**Impact:** any job submitted without a resource request (every REST submit without `-R`) kills the scheduler on its first JOB_UPDATE, and keeps killing it on restart; scheduling stops cluster-wide.
|
||||
**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow)
|
||||
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (src/build.jai)
|
||||
- Code: scheduler-framework @ 6439e585-dirty
|
||||
- Deployment: 3-node HA (xce-deploy-ha) + scheduler + apiserver + one xcecd_virtual worker, single host
|
||||
|
||||
## Reproduction
|
||||
1. Deploy with xce-deploy-ha and `start-all.sh`; `xcecli login -u admin -p admin --url http://localhost:8080`.
|
||||
2. `xcecli jobs submit -J 'arr[1-3]' <script> </dev/null` (no `-R`) and wait for the elements to finish.
|
||||
3. The scheduler dies on the following `schedUpdateJob`; restart it with stderr captured to see the ASAN SEGV above.
|
||||
4. Unit form: `sched_maxtasks_wiring_ingest_test` `test_job_update_with_empty_resreq_uses_default` on a pre-fix tree.
|
||||
- Frequency: deterministic (2/2 live, 1/1 unit, 2026-10-01)
|
||||
- Evidence: /home/claude/tmp/bug-1904/asan-scheduler.log
|
||||
|
||||
## Root cause
|
||||
Both JOB_UPDATE paths (`core_events.c` `xce_core_push_events`, `schedobj.c` update of an existing job) parsed `updated->submit->resreq` as-is. For `""`, `parse_resreq()` zeroes the container (tag 0 = `XCE_RESREQ_PRIMARY`, `u.primary` NULL), parses nothing and returns success; `apply_job_task_count()` (added there by the bug-1773 fix, 41c3694d) then writes `u.primary->minTasks` through NULL. Ingest (`core_build_job`, `make_sched_job`) already maps an empty string to `"type == any"`; the update paths did not.
|
||||
|
||||
## Fix
|
||||
Both update paths now apply the ingest default: an empty resreq string is parsed as `"type == any"`, and `resreq_str` records the string actually parsed.
|
||||
|
||||
## Verification
|
||||
- Test: `test_job_update_with_empty_resreq_uses_default` in `src/scheduler/testsuite/test_maxtasks_wiring_ingest.c`; RED before the fix (the same ASAN SEGV at resreq.c:589), GREEN after
|
||||
- Suites: scheduler meson suite 97/97 OK (asan, src/build.jai)
|
||||
- Live: restarted the fixed scheduler on the same deployment: it replayed the update without crashing and every stuck PEND job (plain, arr2[1], arr2[3]) ran to DONE
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
59
.wolf/bugs/bug-1905.md
Normal file
59
.wolf/bugs/bug-1905.md
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
---
|
||||
id: "bug-1905"
|
||||
status: "open"
|
||||
status_updated: "2026-10-01"
|
||||
status_note: "newly logged"
|
||||
component: "worker"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-01T14:32:18Z"
|
||||
last_seen: "2026-10-01T14:32:18Z"
|
||||
occurrences: 1
|
||||
files: ["src/worker/src/controller_connector.c", "src/worker/src/xcecd_dbus_manager.c"]
|
||||
tags: ["virtual-cluster", "job-control", "kill"]
|
||||
related: []
|
||||
---
|
||||
|
||||
# bug-1905: xcecd_virtual ignores kill of a running job: job keeps RUN until its sleep time elapses
|
||||
|
||||
## Description
|
||||
**Observed:**
|
||||
```
|
||||
14:17:47,625 controller_connector.c:1546 Received job control request to send SIGINFO for job ID: 1-0-1790864259-0
|
||||
14:17:47,625 controller_connector.c:1610 Job control task handled directly for job ID: 1-0-1790864259-0
|
||||
14:27:40,008 xcecd_worker_utils.c:847 Sleep time 600 >= 600 seconds elapsed for worker 1009 (job 1-0-1790864259-0)
|
||||
ctrl-1: on_job_finish job finish jobid 1-0-1790864259-0 exit code 0
|
||||
```
|
||||
`xcecli jobs kill` of a RUN job (plain `sleep 600`, and array elements `arr3[2]`, `arr3[3]`) is accepted (apiserver: `overallStatus accepted`, SIGTERM) and reaches the virtual worker, but the job stays RUN and finishes DONE with exit 0 when its sleep time elapses. Killing a PEND job works (EXIT).
|
||||
|
||||
**Expected:** the virtual worker ends a killed running job as EXITED with exit code 128+signal on its next poll (the `killed_signal` branch in `xcecd_dbus_manager.c:259-262`).
|
||||
**Impact:** kill of running jobs cannot be exercised on virtual-worker deployments (tests, soaks); the real worker path (`ipc_master_send_job_signal`) is not affected by this record.
|
||||
**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow)
|
||||
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (src/build.jai)
|
||||
- Code: scheduler-framework @ 6439e585-dirty
|
||||
- Deployment: 3-node HA (xce-deploy-ha) + scheduler + apiserver + one xcecd_virtual worker, single host
|
||||
|
||||
## Reproduction
|
||||
1. Deploy with xce-deploy-ha and `start-all.sh`; `xcecli login -u admin -p admin --url http://localhost:8080`.
|
||||
2. `xcecli jobs submit -J plain2 sleep 600 </dev/null`; wait until `xcecli jobs list` shows RUN.
|
||||
3. `xcecli jobs kill <id> </dev/null`; `xcecli jobs list -a <id>` still shows RUN, and the job ends DONE after 600 s.
|
||||
- Frequency: deterministic (3/3 kills of running jobs, 2026-10-01)
|
||||
- Evidence: /home/claude/tmp/bug-1905/kill-trace.log
|
||||
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
57
.wolf/bugs/bug-1906.md
Normal file
57
.wolf/bugs/bug-1906.md
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
---
|
||||
id: "bug-1906"
|
||||
status: "open"
|
||||
status_updated: "2026-10-01"
|
||||
status_note: "newly logged"
|
||||
component: "xcecli"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-01T14:32:21Z"
|
||||
last_seen: "2026-10-01T14:32:21Z"
|
||||
occurrences: 1
|
||||
files: ["src/frontend/xcecli/cmd/jobs/list.go"]
|
||||
tags: ["job-array", "error-message", "selection"]
|
||||
related: []
|
||||
---
|
||||
|
||||
# bug-1906: xcecli jobs list drops the selection and its brackets from rejection messages
|
||||
|
||||
## Description
|
||||
**Observed:**
|
||||
```
|
||||
$ xcecli jobs list '1-0-1790863351-0[1,3]'
|
||||
failed to get parameter: [expected <job-id>, <array-id>k or <array-id>a-b]
|
||||
$ xcecli jobs kill '1-0-1790863351-0[1,3]'
|
||||
invalid job identity or element selection '1-0-1790863351-0[1,3]': expected <job-id>, <array-id>[k] or <array-id>[a-b]
|
||||
```
|
||||
`jobs list` passed local selection-parse errors through `request.ExtractErrorMessage`, which strips every `[`/`]` and keeps only the text after the last `": "`.
|
||||
|
||||
**Expected:** the rejection names the selection as written and keeps the `[k]` syntax, as `jobs kill` does (job-array-one-based-index Requirement 5: a rejection names the rejected selection).
|
||||
**Impact:** users of `jobs list` cannot tell which selection was rejected, and the hint shows `<array-id>k` instead of `<array-id>[k]`.
|
||||
**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow)
|
||||
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: none (xcecli is a statically linked Go binary, CGO_ENABLED=0; host glibc 2.42-47)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (src/build.jai)
|
||||
- Code: scheduler-framework @ 6439e585-dirty
|
||||
- Deployment: none needed (client-side parse error); seen against a single-host 3-node HA deploy
|
||||
|
||||
## Reproduction
|
||||
1. `xcecli jobs list 'garbage' </dev/null` (any rejected selection; no server needed).
|
||||
- Frequency: deterministic (4/4 selections, 2026-10-01)
|
||||
- Evidence: not kept (the new test guards it)
|
||||
|
||||
## Root cause
|
||||
`cmd/jobs/list.go` (`RunJobsList`, `runArrayJobStats`) formatted `HandleJobFilter` errors with `request.ExtractErrorMessage`, a sanitizer meant for transport errors: it removes all brackets and keeps only the last `": "`-separated part, which drops the quoted selection.
|
||||
|
||||
## Fix
|
||||
Both sites report the local parse error as written (`failed to get parameter: [%v]`, err), without `ExtractErrorMessage`.
|
||||
|
||||
## Verification
|
||||
- Test: `TestRunJobsListReportsRejectedSelectionVerbatim` in `src/frontend/xcecli/cmd/jobs/list_test.go`; RED before the fix (6 assertion failures), GREEN after
|
||||
- Suites: xcecli `go vet ./...` clean, `go test ./...` all ok (CGO_ENABLED=0); `/xce-build xcecli` asan rc=0
|
||||
- Live: rebuilt xcecli against the deployment: `jobs list` rejections now read e.g. `invalid element selection '1-0-1790863351-0[40000]': element numbers must be at most 32767`
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
Loading…
Add table
Add a link
Reference in a new issue