myxce/.wolf/bugs/bug-1897.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits
bug-1897	fixed	2026-10-01		scheduler	crash	2026-09-30T10:19:59Z	2026-09-30T10:19:59Z	1	src/scheduler/src/plugins/limit.c src/scheduler/src/snapshot_load.c	scheduler sim snapshot limit	bug-1887 bug-1888	0fe27d4b

bug-1897: xce_scheduler --sim segfaults in limit_job_election on a job whose snapshot resreq failed to parse

Description

Observed: replaying data/scheduler-replay-20260928-064842.replay in simulation mode crashes before any decision, 3/3 runs:

[ERROR] [xce_sched_core_load_snapshot] xce_sched_core_load_snapshot parse_resreq failed for jobid 4667606771186008064: Primary: ...
==2723571==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x555555a7cb6e ... T3)
    #0 limit_job_election ../scheduler/src/plugins/limit.c:336
    #1 plugin_job_election ../scheduler/src/plugin.c:601
    #2 sched_allocator ../scheduler/src/sched_election.c:236

The replay's snapshot stores resreqs in the debug-dump format that bug-1888 wrote before its fix. Since the bug-1887 fix, parse_resreq rejects that text, so the snapshot loader logs the error but keeps the job with no schedobj. The allocator thread's job election then reads job->schedobj as NULL in the limit plugin.

Expected: a job whose resreq cannot be parsed from a snapshot is either dropped (with the error logged) or kept pending with a reason, and never reaches election with a NULL schedobj; the simulator must not crash on an old replay.

Impact: crash (SIGSEGV) of xce_scheduler --sim on any replay whose snapshot has unparsable resreqs, including every replay captured before the bug-1888 fix. Suspected: the same path could crash a normal-mode scheduler loading such a snapshot (not tried).

Found by: feature-level validation smoke run of spec scheduler-allocator-runner (2026-09-30); not caused by that feature (limit.c, snapshot_load.c, sched_election.c untouched by it).

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (src/build.alloc, dedicated scheduler-allocator-runner build dir); ASAN_OPTIONS=detect_leaks=0
• Code: scheduler-framework @ 28cff179-dirty
• Deployment: none (simulation mode, xce_scheduler --sim, no controller or workers)

Reproduction

1. git checkout 28cff179 (or later); bash src/tools/xce-build.sh scheduler asan build.alloc --jobs 8
2. cd src && nix develop . --command bash -c 'ASAN_OPTIONS=detect_leaks=0 ./build.alloc/scheduler/xce_scheduler --sim=/home/claude/tmp/bug-1897/scheduler-replay-20260928-064842.replay -l /tmp/sim.log'
3. Observe parse_resreq failed errors in the log, then the ASan SEGV in limit_job_election.

• Frequency: deterministic (3/3 runs, 2026-09-30)
• Evidence: /home/claude/tmp/bug-1897/ (replay file copy, run.out with the ASan report, sim.log)

Root cause

UPDATE (2026-10-01): A job whose snapshot resreq fails to parse is kept by xce_sched_core_load_snapshot() (snapshot_load.c) with resreq and schedobj both NULL, so a running job's restored allocation stays charged. Nothing kept it out of the picker: the loader pushed it with on_job_push(), and in simulation mode sim_bootstrap_picker() (sim_picker.c) re-pushes every core job once the picker exists (bug-1567), which is the path the replay took, since the sim loads its snapshot before any picker is registered. The allocator thread (sched_allocator(), sched_election.c) then ran plugin_job_election() and sched_make_allocation() on the NULL schedobj, and limit_job_election() (limit.c:336) dereferenced it. Exposed since the bug-1887 fix made parse_resreq() reject the debug-dump resreq text that snapshots wrote before the bug-1888 fix, i.e. every replay captured before that fix; the live JOBS_PUSH path was never affected (core_build_job() drops such a job).

Fix

UPDATE (2026-10-01): A job without a schedobj is never made a picker candidate: snapshot_load.c skips on_job_push() for it (logging once that it will not be scheduled) and sim_picker.c's picker rebuild skips it too. The job stays in the core, so its restored allocation is still charged and later events still find it. As a backstop, sched_allocator() (sched_election.c) skips any picked job whose schedobj is NULL, the same way it skips a job it cannot find, so no other route into the picker can crash the allocator thread.

Verification

• Test: test_snapshot_load_leaves_schedobj_null_when_resreq_fails_to_parse (testsuite/test_snapshot_save_load.c, new no-picker-candidate assertion) RED ("Expected 0 Was 1"), GREEN after; test_sched_allocator_skips_job_without_schedobj (testsuite/test_sched_election_null_schedobj.c, new) RED (ASan SEGV in sched_make_allocation, scheduler.c:447), GREEN after; test_snapshot_job_with_unparsable_resreq_is_not_scheduled (testsuite/test_sim_event_loop.c, new) RED (a PEND decision for the job, with the sim_picker.c and sched_election.c changes reverted), GREEN after.
• Suites: scheduler ASAN suite 97/97 (build.alloc), 2026-10-01.
• Live: n/a for HA (no HA path touched). The original replay (scheduler-replay-20260928-064842.replay, --sim) completes all 1511 sessions in 3/3 runs: exit 0, no ASan report, the 50 affected jobs reported once at load as unschedulable, 0 allocator skips.

References

• Related: none