63 lines
3.8 KiB
Markdown
63 lines
3.8 KiB
Markdown
---
|
|
id: "bug-1904"
|
|
status: "fixed"
|
|
status_updated: "2026-10-01"
|
|
status_note: ""
|
|
component: "scheduler"
|
|
severity: "crash"
|
|
timestamp: "2026-10-01T14:32:17Z"
|
|
last_seen: "2026-10-01T14:32:17Z"
|
|
occurrences: 1
|
|
files: ["src/scheduler/src/core_events.c", "src/scheduler/src/schedobj.c"]
|
|
tags: ["resreq", "job-update", "asan", "bug-1773-regression"]
|
|
related: ["bug-1773"]
|
|
fix_commits: ["6e631caa"]
|
|
---
|
|
|
|
# bug-1904: scheduler crashes on a JOB_UPDATE whose resreq is empty: apply_job_task_count writes through NULL
|
|
|
|
## Description
|
|
**Observed:**
|
|
```
|
|
==1580007==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x5c8c80c00867 ...)
|
|
==1580007==The signal is caused by a WRITE memory access.
|
|
#0 apply_job_task_count ../lib/resreq.c:589
|
|
#1 xce_core_push_events ../scheduler/src/core_events.c:1388
|
|
#2 xce_sched_driver_translate_pending_events ../scheduler/src/driver.c:850
|
|
#3 xce_sched_driver_run_normal_mode_round ../scheduler/src/driver.c:1021
|
|
```
|
|
The scheduler process died (stderr was /dev/null under start-all.sh, so it vanished silently) right after the controller pushed `schedUpdateJob` for a REST-submitted job with no `-R`; every later job stayed PEND. Restarting the scheduler crashed again on the same replayed update.
|
|
|
|
**Expected:** an update with an empty resource request is treated as ingest treats it (`"type == any"`) and the scheduler keeps running.
|
|
**Impact:** any job submitted without a resource request (every REST submit without `-R`) kills the scheduler on its first JOB_UPDATE, and keeps killing it on restart; scheduling stops cluster-wide.
|
|
**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow)
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (src/build.jai)
|
|
- Code: scheduler-framework @ 6439e585-dirty
|
|
- Deployment: 3-node HA (xce-deploy-ha) + scheduler + apiserver + one xcecd_virtual worker, single host
|
|
|
|
## Reproduction
|
|
1. Deploy with xce-deploy-ha and `start-all.sh`; `xcecli login -u admin -p admin --url http://localhost:8080`.
|
|
2. `xcecli jobs submit -J 'arr[1-3]' <script> </dev/null` (no `-R`) and wait for the elements to finish.
|
|
3. The scheduler dies on the following `schedUpdateJob`; restart it with stderr captured to see the ASAN SEGV above.
|
|
4. Unit form: `sched_maxtasks_wiring_ingest_test` `test_job_update_with_empty_resreq_uses_default` on a pre-fix tree.
|
|
- Frequency: deterministic (2/2 live, 1/1 unit, 2026-10-01)
|
|
- Evidence: /home/claude/tmp/bug-1904/asan-scheduler.log
|
|
|
|
## Root cause
|
|
Both JOB_UPDATE paths (`core_events.c` `xce_core_push_events`, `schedobj.c` update of an existing job) parsed `updated->submit->resreq` as-is. For `""`, `parse_resreq()` zeroes the container (tag 0 = `XCE_RESREQ_PRIMARY`, `u.primary` NULL), parses nothing and returns success; `apply_job_task_count()` (added there by the bug-1773 fix, 41c3694d) then writes `u.primary->minTasks` through NULL. Ingest (`core_build_job`, `make_sched_job`) already maps an empty string to `"type == any"`; the update paths did not.
|
|
|
|
## Fix
|
|
Both update paths now apply the ingest default: an empty resreq string is parsed as `"type == any"`, and `resreq_str` records the string actually parsed.
|
|
|
|
## Verification
|
|
- Test: `test_job_update_with_empty_resreq_uses_default` in `src/scheduler/testsuite/test_maxtasks_wiring_ingest.c`; RED before the fix (the same ASAN SEGV at resreq.c:589), GREEN after
|
|
- Suites: scheduler meson suite 97/97 OK (asan, src/build.jai)
|
|
- Live: restarted the fixed scheduler on the same deployment: it replayed the update without crashing and every stuck PEND job (plain, arr2[1], arr2[3]) ran to DONE
|
|
|
|
## References
|
|
- Related: none
|