docs(buglog): bug-1904 and bug-1906 fixed in 6e631caa

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-01 14:48:50 +00:00

3.8 KiB

Raw Blame History

id status status_updated status_note component severity timestamp last_seen occurrences files tags related fix_commits

bug-1904

fixed

2026-10-01

scheduler

crash

2026-10-01T14:32:17Z

src/scheduler/src/core_events.c

src/scheduler/src/schedobj.c

resreq

job-update

asan

bug-1773-regression

bug-1773

6e631caa

bug-1904: scheduler crashes on a JOB_UPDATE whose resreq is empty: apply_job_task_count writes through NULL

Description

Observed:

==1580007==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x5c8c80c00867 ...)
==1580007==The signal is caused by a WRITE memory access.
    #0 apply_job_task_count ../lib/resreq.c:589
    #1 xce_core_push_events ../scheduler/src/core_events.c:1388
    #2 xce_sched_driver_translate_pending_events ../scheduler/src/driver.c:850
    #3 xce_sched_driver_run_normal_mode_round ../scheduler/src/driver.c:1021

The scheduler process died (stderr was /dev/null under start-all.sh, so it vanished silently) right after the controller pushed schedUpdateJob for a REST-submitted job with no -R; every later job stayed PEND. Restarting the scheduler crashed again on the same replayed update.

Expected: an update with an empty resource request is treated as ingest treats it ("type == any") and the scheduler keeps running. Impact: any job submitted without a resource request (every REST submit without -R) kills the scheduler on its first JOB_UPDATE, and keeps killing it on restart; scheduling stops cluster-wide. Found by: spec task (job-array-one-based-index task 10.2 live REST flow)

Environment

OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
Build: asan (src/build.jai)
Code: scheduler-framework @ 6439e585-dirty
Deployment: 3-node HA (xce-deploy-ha) + scheduler + apiserver + one xcecd_virtual worker, single host

Reproduction

Deploy with xce-deploy-ha and start-all.sh; xcecli login -u admin -p admin --url http://localhost:8080.
xcecli jobs submit -J 'arr[1-3]' <script> </dev/null (no -R) and wait for the elements to finish.
The scheduler dies on the following schedUpdateJob; restart it with stderr captured to see the ASAN SEGV above.
Unit form: sched_maxtasks_wiring_ingest_test test_job_update_with_empty_resreq_uses_default on a pre-fix tree.

Frequency: deterministic (2/2 live, 1/1 unit, 2026-10-01)
Evidence: /home/claude/tmp/bug-1904/asan-scheduler.log

Root cause

Both JOB_UPDATE paths (core_events.c xce_core_push_events, schedobj.c update of an existing job) parsed updated->submit->resreq as-is. For "", parse_resreq() zeroes the container (tag 0 = XCE_RESREQ_PRIMARY, u.primary NULL), parses nothing and returns success; apply_job_task_count() (added there by the bug-1773 fix, 41c3694d) then writes u.primary->minTasks through NULL. Ingest (core_build_job, make_sched_job) already maps an empty string to "type == any"; the update paths did not.

Fix

Both update paths now apply the ingest default: an empty resreq string is parsed as "type == any", and resreq_str records the string actually parsed.

Verification

Test: test_job_update_with_empty_resreq_uses_default in src/scheduler/testsuite/test_maxtasks_wiring_ingest.c; RED before the fix (the same ASAN SEGV at resreq.c:589), GREEN after
Suites: scheduler meson suite 97/97 OK (asan, src/build.jai)
Live: restarted the fixed scheduler on the same deployment: it replayed the update without crashing and every stuck PEND job (plain, arr2[1], arr2[3]) ran to DONE

References

Related: none

3.8 KiB Raw Blame History