myxce/.wolf/bugs/bug-1763.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	legacy
bug-1763	fixed	2026-09-30		scheduler	crash	2026-09-02T13:00:00Z	2026-09-19T09:00:00Z	1	src/scheduler/src/sim_loop.c (sched_sim_driver_run) src/scheduler/src/sim.c (sched_sim_driver_stop_threads, sched_sim_driver_fini) src/scheduler/testsuite/test_sim_record_output.c	scheduler sim repl asan stack-use-after-return flaky thread-lifecycle scheduler-sim-repl-verbosity	bug-1598 bug-1632	cfa0a941	true

bug-1763: sched_sim_record_output_test (scheduler-sim-repl-verbosity task 3.2 GREEN-phase build)…

Description

sched_sim_record_output_test (scheduler-sim-repl-verbosity task 3.2 GREEN-phase build): AddressSanitizer stack-use-after-return in cbuf_get (lib/mylock.c:151), thread T1 (sim_picker_thread_fn, sim_loop.c:456), reading the stack of test_output_file_is_decodable_with_full_stream_symmetry's stack-local sim (xce_sched_sim_driver_t, test_sim_record_output.c:341) after that frame returned. Fired once during a /xce-build scheduler --incremental run after this task's repl_break.c edits; 4/4 direct re-runs of the isolated binary afterward were clean (OK).

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan; meson runs these tests with ASAN_OPTIONS=detect_leaks=0
• Code: original sightings on scheduler-framework in Aug/Sep 2026 (commit not recorded); root-caused and fixed on 2026-09-29 at dc94da20
• Deployment: n/a -- unit test binary, no cluster

Reproduction

1. bash src/tools/xce-build.sh scheduler (full suite under load); the ASAN abort shows in sched_sim_record_output_test's log.
2. Deterministic check of the mechanism: test_run_returns_with_its_threads_stopped (src/scheduler/testsuite/test_sim_verify.c) -- sched_sim_driver_run() returns with allocator_thread_live still true.

• Frequency: original ASAN report rare (1 in several full-suite runs; 0/30 isolated runs, 2026-09-29); the mechanism check is deterministic
• Evidence: not kept -- the original build logs were not saved; the test above reproduces the cause

Root cause

sched_sim_driver_run() (sim_loop.c) starts the picker and allocator threads (sched_job_picker/sched_allocator, or a test's stand-ins) on &sim->core and returned with them still running; only sched_sim_driver_fini() (sim.c) cancelled and joined them. main.c pairs the two, but 24 scheduler test files (35 call sites) run the driver on a stack-local xce_sched_sim_driver_t and never call fini. The threads, parked in cbuf_get() on the core's cbufs, then held a pointer into a dead frame; when a later test reused that stack memory and something woke them, one read it -- the reported stack-use-after-return in cbuf_get (mylock.c). Timing-dependent, so seen only occasionally under full-suite load. Also: if the allocator thread failed to start, the already-running picker was never stopped at all.

Fix

src/scheduler/src/sim_loop.c: the loop moved into static sim_driver_run_body(); sched_sim_driver_run() calls it and then sched_sim_driver_stop_threads() on every return path, so no thread outlives the run; an allocator start failure now cancels and joins the running picker. src/scheduler/src/sim.c: new sched_sim_driver_stop_threads() (cancel + join both, idempotent via allocator_thread_live), used by the run and by sched_sim_driver_fini(); it no longer pushes a jobid 0 to wake the allocator -- cbuf waits in pthread_cond_wait (a cancellation point, mutex released by a cleanup handler), and the fake end-of-session was visible to the allocator (test_sim_discrete_clock caught it as an extra session). sim.h documents the guarantee.

Verification

• Test: test_run_returns_with_its_threads_stopped in src/scheduler/testsuite/test_sim_verify.c runs a replay through sched_sim_driver_run() on a stack-local driver and asserts no thread is left running: RED before the fix (bug-1632: sched_sim_driver_run() returned with its picker and allocator threads still running), GREEN after -- deterministic, unlike the original ASAN report
• Suites: scheduler 90/90 test binaries, asan, 0 sanitizer reports (first attempt had test_sim_discrete_clock fail on the jobid-0 wake-up; fixed as above)
• Live: n/a, test-harness lifetime bug; 30 isolated runs each of sched_sim_verify_test and sched_sim_record_output_test (ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=1) were clean both before and after the fix, matching the records' note that it never reproduced in isolation

References

• Fix commit: cfa0a941
• Related: bug-1632 (same cause, other test binary), bug-1884 (added sched_sim_driver_fini)