myxce/.wolf/bugs/bug-1904.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related
bug-1904	open	2026-10-01	newly logged	scheduler	crash	2026-10-01T14:32:17Z	2026-10-01T14:32:17Z	1	src/scheduler/src/core_events.c src/scheduler/src/schedobj.c	resreq job-update asan bug-1773-regression	bug-1773

bug-1904: scheduler crashes on a JOB_UPDATE whose resreq is empty: apply_job_task_count writes through NULL

Description

Observed:

==1580007==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x5c8c80c00867 ...)
==1580007==The signal is caused by a WRITE memory access.
    #0 apply_job_task_count ../lib/resreq.c:589
    #1 xce_core_push_events ../scheduler/src/core_events.c:1388
    #2 xce_sched_driver_translate_pending_events ../scheduler/src/driver.c:850
    #3 xce_sched_driver_run_normal_mode_round ../scheduler/src/driver.c:1021

The scheduler process died (stderr was /dev/null under start-all.sh, so it vanished silently) right after the controller pushed schedUpdateJob for a REST-submitted job with no -R; every later job stayed PEND. Restarting the scheduler crashed again on the same replayed update.

Expected: an update with an empty resource request is treated as ingest treats it ("type == any") and the scheduler keeps running. Impact: any job submitted without a resource request (every REST submit without -R) kills the scheduler on its first JOB_UPDATE, and keeps killing it on restart; scheduling stops cluster-wide. Found by: spec task (job-array-one-based-index task 10.2 live REST flow)

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (src/build.jai)
• Code: scheduler-framework @ 6439e585-dirty
• Deployment: 3-node HA (xce-deploy-ha) + scheduler + apiserver + one xcecd_virtual worker, single host

Reproduction

1. Deploy with xce-deploy-ha and start-all.sh; xcecli login -u admin -p admin --url http://localhost:8080.
2. xcecli jobs submit -J 'arr[1-3]' <script> </dev/null (no -R) and wait for the elements to finish.
3. The scheduler dies on the following schedUpdateJob; restart it with stderr captured to see the ASAN SEGV above.
4. Unit form: sched_maxtasks_wiring_ingest_test test_job_update_with_empty_resreq_uses_default on a pre-fix tree.

• Frequency: deterministic (2/2 live, 1/1 unit, 2026-10-01)
• Evidence: /home/claude/tmp/bug-1904/asan-scheduler.log

Root cause

Both JOB_UPDATE paths (core_events.c xce_core_push_events, schedobj.c update of an existing job) parsed updated->submit->resreq as-is. For "", parse_resreq() zeroes the container (tag 0 = XCE_RESREQ_PRIMARY, u.primary NULL), parses nothing and returns success; apply_job_task_count() (added there by the bug-1773 fix, 41c3694d) then writes u.primary->minTasks through NULL. Ingest (core_build_job, make_sched_job) already maps an empty string to "type == any"; the update paths did not.

Fix

Both update paths now apply the ingest default: an empty resreq string is parsed as "type == any", and resreq_str records the string actually parsed.

Verification

• Test: test_job_update_with_empty_resreq_uses_default in src/scheduler/testsuite/test_maxtasks_wiring_ingest.c; RED before the fix (the same ASAN SEGV at resreq.c:589), GREEN after
• Suites: scheduler meson suite 97/97 OK (asan, src/build.jai)
• Live: restarted the fixed scheduler on the same deployment: it replayed the update without crashing and every stuck PEND job (plain, arr2[1], arr2[3]) ran to DONE

References

• Related: none