50 lines
4.8 KiB
Markdown
50 lines
4.8 KiB
Markdown
---
|
|
id: "bug-1763"
|
|
status: "fixed"
|
|
status_updated: "2026-09-30"
|
|
status_note: ""
|
|
component: "scheduler"
|
|
severity: "crash"
|
|
timestamp: "2026-09-02T13:00:00Z"
|
|
last_seen: "2026-09-19T09:00:00Z"
|
|
occurrences: 1
|
|
files: ["src/scheduler/src/sim_loop.c (sched_sim_driver_run)", "src/scheduler/src/sim.c (sched_sim_driver_stop_threads, sched_sim_driver_fini)", "src/scheduler/testsuite/test_sim_record_output.c"]
|
|
tags: ["scheduler", "sim", "repl", "asan", "stack-use-after-return", "flaky", "thread-lifecycle", "scheduler-sim-repl-verbosity"]
|
|
related: ["bug-1598", "bug-1632"]
|
|
fix_commits: ["cfa0a941"]
|
|
legacy: true
|
|
---
|
|
|
|
# bug-1763: sched_sim_record_output_test (scheduler-sim-repl-verbosity task 3.2 GREEN-phase build)…
|
|
|
|
## Description
|
|
sched_sim_record_output_test (scheduler-sim-repl-verbosity task 3.2 GREEN-phase build): AddressSanitizer stack-use-after-return in cbuf_get (lib/mylock.c:151), thread T1 (sim_picker_thread_fn, sim_loop.c:456), reading the stack of test_output_file_is_decodable_with_full_stream_symmetry's stack-local `sim` (xce_sched_sim_driver_t, test_sim_record_output.c:341) after that frame returned. Fired once during a `/xce-build scheduler --incremental` run after this task's repl_break.c edits; 4/4 direct re-runs of the isolated binary afterward were clean (OK).
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan; meson runs these tests with ASAN_OPTIONS=detect_leaks=0
|
|
- Code: original sightings on scheduler-framework in Aug/Sep 2026 (commit not recorded); root-caused and fixed on 2026-09-29 at dc94da20
|
|
- Deployment: n/a -- unit test binary, no cluster
|
|
|
|
## Reproduction
|
|
1. `bash src/tools/xce-build.sh scheduler` (full suite under load); the ASAN abort shows in sched_sim_record_output_test's log.
|
|
2. Deterministic check of the mechanism: test_run_returns_with_its_threads_stopped (src/scheduler/testsuite/test_sim_verify.c) -- sched_sim_driver_run() returns with allocator_thread_live still true.
|
|
- Frequency: original ASAN report rare (1 in several full-suite runs; 0/30 isolated runs, 2026-09-29); the mechanism check is deterministic
|
|
- Evidence: not kept -- the original build logs were not saved; the test above reproduces the cause
|
|
|
|
## Root cause
|
|
sched_sim_driver_run() (sim_loop.c) starts the picker and allocator threads (sched_job_picker/sched_allocator, or a test's stand-ins) on &sim->core and returned with them still running; only sched_sim_driver_fini() (sim.c) cancelled and joined them. main.c pairs the two, but 24 scheduler test files (35 call sites) run the driver on a stack-local xce_sched_sim_driver_t and never call fini. The threads, parked in cbuf_get() on the core's cbufs, then held a pointer into a dead frame; when a later test reused that stack memory and something woke them, one read it -- the reported stack-use-after-return in cbuf_get (mylock.c). Timing-dependent, so seen only occasionally under full-suite load. Also: if the allocator thread failed to start, the already-running picker was never stopped at all.
|
|
|
|
## Fix
|
|
src/scheduler/src/sim_loop.c: the loop moved into static sim_driver_run_body(); sched_sim_driver_run() calls it and then sched_sim_driver_stop_threads() on every return path, so no thread outlives the run; an allocator start failure now cancels and joins the running picker. src/scheduler/src/sim.c: new sched_sim_driver_stop_threads() (cancel + join both, idempotent via allocator_thread_live), used by the run and by sched_sim_driver_fini(); it no longer pushes a jobid 0 to wake the allocator -- cbuf waits in pthread_cond_wait (a cancellation point, mutex released by a cleanup handler), and the fake end-of-session was visible to the allocator (test_sim_discrete_clock caught it as an extra session). sim.h documents the guarantee.
|
|
|
|
## Verification
|
|
- Test: test_run_returns_with_its_threads_stopped in src/scheduler/testsuite/test_sim_verify.c runs a replay through sched_sim_driver_run() on a stack-local driver and asserts no thread is left running: RED before the fix (`bug-1632: sched_sim_driver_run() returned with its picker and allocator threads still running`), GREEN after -- deterministic, unlike the original ASAN report
|
|
- Suites: scheduler 90/90 test binaries, asan, 0 sanitizer reports (first attempt had test_sim_discrete_clock fail on the jobid-0 wake-up; fixed as above)
|
|
- Live: n/a, test-harness lifetime bug; 30 isolated runs each of sched_sim_verify_test and sched_sim_record_output_test (ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=1) were clean both before and after the fix, matching the records' note that it never reproduced in isolation
|
|
|
|
## References
|
|
- Fix commit: cfa0a941
|
|
- Related: bug-1632 (same cause, other test binary), bug-1884 (added sched_sim_driver_fini)
|